Einstein Analytics: Precision and Scale

Precision and Scale are important and required for computeExpression node that returns Numeric in Dataflow, otherwise, your dataflow rum will fail.

For numeric, as per this article External Data Metadata Format Reference

• precision: the maximum number of digits in a numeric value, includes all numbers to the left and to the right of the decimal point (but excludes the decimal point character). Value can be up to 18.
• scale: the number of digits to the right of the decimal point in a numeric value, must be less than the precision value.

But in short:
  precision: must be 1 - 18
  scale: must be 0 - 17 and less than the precision value


Let us see how this works in reality. I'll do a few same calculations on computeExpression, but with different precision and scale, the formula is A/B for all calculations, here is the result:

Calc_10_5 mean, precision = 10 and scale = 5, and etc. At a glance, you may think that all decimal points do not exist, this is incorrect as you need to "format numbers" on the widget or metadata.

For this blog testing, I set 5 digits decimal point:

Here is the result after all fields set with 5 decimal points:

From the above table, "scale" shows the difference in the calculation result, the result will be round up or round down based on the decimal point defined in the scale.

Notice that decimal point "below 0.5" will be round down, while "0.5 and above" will be round up. But, if scale = 0, all decimal points will be round down, see calc_10_0.



Reference: External Data Metadata Format Reference