Repository navigation
Why row-wise statistics slightly differ from numpy's one? #1684
Description
If I apply sum(), mean(), stddev() etc. functions column wise I have exactly the same results as from numpy, but when I apply them row wise results are slightly different.
Example for row-wise:
kvs@lair:build$ python3 ../src/np-stat.py /tmp/a2.csv 1
sum : [-53.862728520223981832 26.572322101126204075]
mean : [-0.005386272852022398 0.002657232210112620]
stddev : [1.002509016432828526 0.995021343531496738]
variance: [1.005024328029117120 0.990067474083224819]
kvs@lair:build$ ./tests/test2 /tmp/a2.csv 1
sum : [-53.862728520223711826 26.572322101126072624 ]
mean : [-0.005386272852022371 0.002657232210112607 ]
stddev : [1.002509016432829636 0.995021343531495850 ]
variance: [1.005024328029119340 0.990067474083223042 ]
Code for C++ & xtensor:
#include <cstdio>
#include <cstdlib>
#include <fstream>
#include <xtensor/xarray.hpp>
#include <xtensor/xio.hpp>
#include <xtensor/xcsv.hpp>
#include <xtensor/xmath.hpp>
template <typename T>
void
print(const char* tag, const T& values)
{
std::printf("%s [", tag);
for (const auto& v : values) {
std::printf("%.18f ", v);
}
std::printf("]\n");
}
int
main(int argc, char** argv)
{
if (argc < 3) {
std::printf("Usage: %s <file> <axis>\n", argv[0]);
std::printf("Where:\n");
std::printf(" file -- file with data matrix in CSV format\n");
std::printf(" axis -- which axis to use to calculate statistics (0 -- y, 1 -- x)\n");
return EXIT_FAILURE;
}
auto axis = std::atoi(argv[2]);
std::fstream s(argv[1], std::ios::in);
if (!s) {
std::fprintf(stderr, "Error: couldn't open file '%s'\n", argv[1]);
return EXIT_FAILURE;
}
auto data = xt::load_csv<double>(s);
auto sum = xt::sum(data, {axis});
auto mean = xt::mean(data, {axis});
auto stddev = xt::stddev(data, {axis});
auto variance = xt::variance(data, {axis});
print("sum :", sum);
print("mean :", mean);
print("stddev :", stddev);
print("variance:", variance);
return 0;
}Code for numpy:
import sys
import numpy as np
def main():
if len(sys.argv) < 3:
print("{} <file> <axis>".format(argv[0]))
sys.exit(0)
data = np.loadtxt(sys.argv[1], dtype=np.float64, delimiter=',')
axis = int(sys.argv[2])
stddev = np.std(data, axis)
sum = np.sum(data, axis)
mean = np.mean(data, axis)
variance = np.var(data, axis)
np.set_printoptions(formatter={'float': lambda x: "{0:0.18f}".format(x)})
print("sum : {}".format(sum))
print("mean : {}".format(mean))
print("stddev : {}".format(stddev))
print("variance: {}".format(variance))
if __name__ == "__main__":
main()Used numpy: 1.13.3-2ubuntu1
I am also attaching data I used for this test.
a2.csv.gz
Activity
JohanMabille commentedon Aug 27, 2019
The summation algorithms used in Numpy and xtensor are a bit different. Numpy uses partial pairwise summation when it can while xtensor does not. Besides, even if we used similar approaches, the order in which you perform the operations may change the result (although it tends to be smaller with the pairwise summation).