What is the totalization error of fp16?

The address for this article:/fp16-err/

Recently in the project need to implement the fp16 data type to do the calculation of FFN, arithmetic implementation of the students feedback error and x86 on the golden data obtained from a relatively large error. At first, I thought it was a problem of doing numerical simulation on the x86 side. Later also realized a comparison, found that the cumulative error is indeed quite large.

Comparison of measured results

int main()
{
    // Seed with a real random value, if available
    std::random_device rd;
    std::mt19937 gen(rd());
    std::uniform_real_distribution<> dist(0, 0.01);
    
    float16_t lhs[4096] = {0};
    float16_t rhs[4096] = {0};
    for (int i = 0; i < 4096; i++) {
        lhs[i] =  dist(gen);
        rhs[i] =  dist(gen);
    }
    float16_t res_fp16 = 0;
    float res_fp32 = 0;

    for (int i = 0; i < 4096; i++) {
        res_fp16 += lhs[i] * rhs[i];
        res_fp32 += lhs[i] * rhs[i];
    }
    std::cout << "fp16 " << res_fp16 << std::endl;
    std::cout << "fp32 " << res_fp32 << std::endl;
    wirte2file("/data/local/tmp/lhs", reinterpret_cast<char*>(lhs), 8192);
    wirte2file("/data/local/tmp/rhs", reinterpret_cast<char*>(rhs), 8192);
}

Result Output:

fp16 0.0942383
fp32 0.103176

The relative error is up to 8.1%. No wonder the feedback is problematic.

dim	absolute error
100	1.63913e-07
1000	-0.00033829
2000	-0.000909835
4000	-0.00924221

golden Where do data errors come from?

The actual generation of golden data also takes into account the effect of differences in value types, so why is there still an error?

Comparing dot's line of sight to the direct cumulative result

import numpy as np
import torch

lhs = ("lhs",dtype=np.float16)
rhs = ("rhs",dtype=np.float16)

lhs = torch.from_numpy(lhs)
rhs = torch.from_numpy(rhs)

res = ([1]).half()
res[0] = 0
for i in range(4096):
    res += lhs[i:i+1] * rhs[i:i+1]

print(res)
print((lhs, rhs))

tensor([0.0942], dtype=torch.float16)
tensor(0.1041, dtype=torch.float16)

The result is correct. torch's dot implementation probably uses a higher value type for the accumulation.