1,381 questions
Best practices
0
votes
8
replies
81
views
Does float16 Round After Each Operation or Only at the End? A Code Investigation
I have this question that I want to test in code: if we have x1, x2, x3, x4 (all float16) and f() = (x1 + x2) * x3 - x4, I want to know whether the machine rounds after each operation or calculates ...
4
votes
1
answer
283
views
Accurate computation of the inverse gamma function with the standard C math library
The inverse of the gamma function over the reals is multivalued with an infinite number of branches. This self-answered question is about the principal
inverse of the gamma function, Γ0-1(x), whose ...
- 27.4k
0
votes
1
answer
324
views
Inaccuracy replicating Fortran mixed-precision expression in Rust
I have the following code in my Fortran program, where both a and b are declared as REAL (KIND=8):
a = 0.12497443596150659d0
b = 1.0 + 0.00737 * a
This yields b as 1.0009210615647672
For comparison, ...
- 356
1
vote
0
answers
101
views
How does R round(1.535, 2) → 1.53? [duplicate]
My understanding is that R uses IEC 60559 rounding with "Round to nearest, ties to even", but it does not appear to be generating valid results. For decimal part .535 rounded to 2 places, my ...
- 1,711
1
vote
1
answer
125
views
integer exact computation with logs
I need to compute ceil(log_N(i)) where log_N is the log with positive integer base N and i is also a positive integer.
The straight forward python implementation using floating point math fails at ...
- 2,156
1
vote
0
answers
88
views
Floating point accuracy changes based on how values are initialised [duplicate]
If I run x=0:0.1:1; x==0.2, I get equality at the third element (as expected). However, if I change it to x=-1:0.1:1; x==0.2 I no longer get any hits. Why the change? (I am familiar with some details ...
- 639
0
votes
1
answer
64
views
Cutover point between two CDF implementations
My goal is to implement a CDF (cumulative distribution function) for the Standard Normal Distribution and to do so with the best possible numeric accuracy.
This is my starting point: (1 + erf(x / sqrt(...
- 230k
4
votes
1
answer
125
views
How to compute sign(a² - b * c) * sqrt(abs(a² - b * c)) accurately with floating point arithmetics?
Question
I was wondering if there is a numerically accurate way to compute
sign(a² - b * c) * sqrt(abs(a² - b * c))
with floating point arithmetics because it suffers from (ordered from most to least ...
- 141
1
vote
0
answers
111
views
Calculate accuracy in libtorch (C++) when using Softmax
The below code is able to learn the generic pattern and it is able to predict as expected:
torch::manual_seed(42);
torch::nn::Sequential net(
torch::nn::Linear(300, 301),
torch::nn::...
0
votes
3
answers
176
views
Why does Double.CompareTo() use "==" internally?
One should not use == to compare real numbers, every C# developer knows that.
Then why the Double.CompareTo method, from the framework, is implemented this way for checking equality?
public int ...
- 550
-1
votes
5
answers
287
views
Format floating-point numbers with sed to equal precision - add trailing zeros
Is it possible to format floating point numbers with sed to extend to equal precision of e.g. 8 digits after the decimal point?
Because this problem is part of a bigger context, it is essential to use ...
3
votes
2
answers
258
views
Function which returns coefficients of Legendre Polynomials not accurate enough
I'm trying to write a function which returns the coefficients of the nth legendre polynomial.
// store previous polynomials to make the recursive process faster
unordered_map<int, vector<double&...
- 293
0
votes
3
answers
207
views
How to improve precision when using float type to compute?
I encountered a problem. The code is as follows:
float a = 2.f;
float t = 0.0000025f;
float b = a + t;
float c = b - a;
printf("b = %.8f\n", b);
printf("c = %.8f\n", c);
The ...
0
votes
2
answers
105
views
Matplotlib Makes Trippy Incorrect plots
I fill a numpy array with 0 or 1 depending if a computed array indexed value is more or less than a given value. This results in an array of zeros and ones. Matplotlib doesnt think so
The comparison ...
- 61
4
votes
4
answers
188
views
How can I solve Float type Machine Epsilon Calculation Error?
#include <stdio.h>
void get_eps(float *eps) {
*eps = 1.0f;
while ((1.0f + *eps / 2.0f) != 1.0f) {
*eps /= 2.0f;
}
}
int main(){
float eps;
get_eps(&eps);
...
- 41
0
votes
0
answers
79
views
Reading Float data with float variable in stored procedure rounds numbers
In SQL Server Management Studio:
DECLARE @Amount FLOAT
SELECT SUM(Amount)
FROM Docs
WHERE DocID IN (104482, 104483, 104484, 104486, 104489, 104491, 104493)
-- displays in results window 178457,05
...
- 11
5
votes
1
answer
224
views
Accurate computation of the inverse of the scaled complementary error function using standard C
This is a self-answered question, as encouraged on Stackoverflow for sharing knowledge. As such, it is a follow-on to my previous self-answered question about the accurate computation of the ...
- 27.4k
3
votes
1
answer
289
views
Accurate computation of the negative branch of the real-valued Lambert W function in standard C
This is a self-answered question, as encouraged on Stackoverflow for sharing knowledge. As such, it is a follow-on to my previous self-answered question regarding accurate computation of the principal ...
- 27.4k
2
votes
1
answer
465
views
How does rounding works in float multiplication?
The exact value of float 0.7f is 0.69999...
so I thought the result of 0.7f * 100f would be something below 70, like 69.99999...
But the result is exact 70.
Does float multiplication involve such ...
- 23
2
votes
1
answer
521
views
Can Neumaier summation be sped up?
Neumaier summation is an improvement of Kahan summation for accurately summing arrays of floats.
import numba as nb
@nb.njit
def neumaier_sum(arr):
s = arr[0]
c = 0.0
for i in range(1, ...
- 21.6k
0
votes
1
answer
95
views
How does SQL Server store integer values of over 2^24 in a FLOAT(53) column?
I have a SQL Server database from which I can read integer values of (way) over 2^24 from a column that has datatype float. This confuses me since these values should be to big/lengthy to be ...
- 125
-1
votes
1
answer
83
views
Float calculation problem. 12.7 - 20 + 7.3 not equal zero [duplicate]
12.7 - 20 + 7.3 = -8.8817841970013e-016
I got this problem when I read Programming in Lua book, and I cannot figure it out how to get zero as the result of the calculation.
When I formatted the result ...
0
votes
1
answer
104
views
How much accuracy is lost due to representing time as a IEEE 754 binary64 number?
My python program has a busy loop. Within the loop, some delays are calculated in the following fashion:
timeout_s: float = args.timeout_s
timeout_end_s: float = time.time() + timeout_s
while True:
...
- 147k
0
votes
1
answer
175
views
ceil(double(int)*double) or ceil(float(int*double))?
I had this C++ code:
int x;
double y;
// do sth
int z = ceil(x*y);
If x = 410, y = 1.1, then z = 452 when it should've been 451. I changed to z = ceil(float(x*y)) and it works. My friend suggests z ...
- 13
4
votes
5
answers
1k
views
Fast implementation of complementary error function with 5-digit accuracy
[While this is a self-answered question, I will happily upvote and accept any alternative answer that either provides superior accuracy for the same amount of computational work or reduces the amount ...
- 27.4k
0
votes
0
answers
39
views
R: floating point errors in simple math operations [duplicate]
So of course:
1023.1 - 0.1 < 1023
[1] FALSE
but...
1024.1 - 0.1 < 1024
[1] TRUE
Is this a bug? Is it expected behaviour?
Is there some way I can avoid this, such as rounding all my numbers to ...
- 292
0
votes
1
answer
80
views
Truncating decimal digits like Excel in python
In my excel file, I've some calculations. Calculation returns 0,345 and excel convert it to 0,35. I made it in python. Python is returning 0,345, but I want to take 0,35 like Excel. I've tried with ...
user22304060
0
votes
1
answer
105
views
Calculating power of (1-1/x)^y for very large x,y
I am thinking about calculating $(1-1/x)^y$ for very large x,y numbers in python.
One way todo so, it to use exp(y*log1p(-1/x)), but I am not sure about its accuracy.
Are they any results on using ...
- 367
0
votes
0
answers
82
views
Why assignment changes the precision of calculation of MATLAB?
I am writing some scripts using MATLAB R2020a and I am having a problem: there are two values I want to assign to a vector z_samples, namely x_samples(l_indexes)./sum(x_samples)(a vector) and ...
0
votes
1
answer
188
views
Rounding errors when computing intersection between two lines in python
I'm attempting to create a Python script that identifies all the unique intersection points, eliminating duplicates, among all the lines passing through a subset of points of a regular grid in 2D or ...
- 135
1
vote
2
answers
745
views
Workarounds for High Floating-Point Error in NumPy Arrays Raised to Powers
I have data that represents responses from a 24(?)-bit A/D converter. I am setting up the data to go into a polynomial regression in the function below, but I am getting detrimental amounts of ...
- 41
1
vote
2
answers
254
views
imshow plotting very large integers, but "dtype object cannot be converted to float"
I have the following code, plotting a function on a grid, where the function happens to have a very large integer value:
import matplotlib.pyplot as plt
from matplotlib.ticker import ScalarFormatter, ...
- 127
0
votes
1
answer
1k
views
Conversion accuracy issues of E57 to LAS in Python using pye57 and laspy
I am attempting to convert an .e57 pointcloud into a .las pointcloud but I am running into issues which seem to be caused by accuracy. The following images illustrate the original .e57 in CloudCompare,...
- 609
1
vote
3
answers
107
views
Without overflow and underflow, for a float number x, is x/4 always equals to x/2/2 (also for x*4 == x*2*2)?
For example, as I know, a float point x, which x/n^2 may not be equal to x/n/n, because in x/n/n, which would create an intermediate x/n first. For example:
document.write(1.23456789/49 == 1....
6
votes
4
answers
312
views
How to properly round to the nearest integer in double-double arithmetic
I have to analyse a large amount of data using Python3 (PyPy implementation), where I do some operations on quite large floats, and must check if the results are close enough to integers.
To exemplify,...
- 337
1
vote
0
answers
195
views
Different floating point precision in Matrix multiplication in Python and C++
I am trying to do a very simple thing
take a square shape 2d array
take its transpose
multiply the array with its transpose
I am trying to perform the above steps in C++ and Python as shown in the ...
- 5,789
0
votes
1
answer
154
views
Why explicit Integer conversion doesn't apply as expected
Please be aware that my question is not "Why do floating numbers lose precision" I am aware that not all fractional parts of a number can sit in binary form. What's really stored is actually ...
- 23
3
votes
1
answer
119
views
python rounding error when sampling variable Y as a function of X with histogram?
I'm trying to sample a variable (SST) as a function of another variable (TCWV) using the function histogram, with weights set to the sample variable like this:
# average sst over bins
num, _ = np....
- 8,206
3
votes
3
answers
849
views
What are two numbers of 16 decimal digits precision that map to the same double-precision float?
It is said that any decimal number expressed with 15 digits of precision within the range covered by double-precision floats can be uniquely identified with a particular float. But, that if we include ...
- 1,205
1
vote
1
answer
100
views
Exponential limits for C and Matlab
In computers, the math operation of floating numbers is actually processed using the base number and the exponential number separately, and then combine them together. We learn this in our computation ...
- 37
2
votes
3
answers
322
views
Equality of floating point numbers after storing/loading/moving
A coworker and I have a disagreement about what can happen when comparing two floating-point numbers that have not been mathematically operated on. i.e. the numbers may have been moved around memory ...
- 761
0
votes
2
answers
606
views
Rendering Accuracy Problem with Large Zoom in/Out Range in OpenGL Graphics
I have a simple camera based 'modern' OpenGL 3D graphics display, for rendering relatively simple objects that are constructed from collections of specified points, lines and curves (e.g. a cube, ...
- 107
1
vote
2
answers
266
views
float imprecision in C
I'm starting to learn c programming language.
I know that floats are represented by having:
1 bit for sign
8 bits for exponent
23 bits for mantissa
Does this mean that I can represent precisely only ...
- 2,325
-4
votes
1
answer
325
views
Multiplication of large numbers python
I decided to multiply two numbers:
1.0
17299991019999108999910899991119999
and as a result , I received in response:
1.729999101999911e+34
Code:
print(1.0*17299991019999108999910899991119999)
1....
user19345539
0
votes
1
answer
183
views
Lua float precision messed up?
Disclaimer: I'm using lua on the table top simulator game, I don't know if this interfere in any kind of way with the "regular work" of lua.
I already have a bunch of vectors pre-calculated:
...
1
vote
1
answer
66
views
Program giving inaccurate value
The below program calculates 262 + 261 + 260 + ... + 21 + 20.
Using double type to store sum:
double sum = 0;
for (int i = 0; i < 63; i++) {
sum += pow(2.0, i);
// print("i : $i sum :...
user8234870
0
votes
0
answers
24
views
I know the max decimal place will be 18, if I user ini_set('precision', 18), it changes the number output again [duplicate]
`ini_set('precision', 18);
$number2 = 5.00025051541321;
echo ($number2);
Output: 5.00025051541321019`
What can be the issue and if someone help me to solve that.
1
vote
0
answers
220
views
Alternatives to == in dplyr::filter, to accomodate floating point numbers
First off, let me say that I am aware that we are constrained by the limitations of computer arithmetic and floating point numbers and that 0.8 doesn't equal 0.8, sometimes. I'm curious about ways to ...
- 11
0
votes
1
answer
61
views
Will Math.round(intA/intB) always return the same on every device?
I'm worried that something like 7/2 might become 3.49999999 instead of 3.5 on some computer then round to 3 instead of 4. Is that possible?
I need every computer to re-run the same math exactly. That'...
- 2,545
3
votes
1
answer
99
views
For any float number x,y,integer n, where const y=n-x, and 0<=x,y,n<=MAX_VALUE, is x+y==n always true?
For example:
const n=5;
const x=1.23;
const y=n-x;
document.write(x+y==n);
the code about prints true for n=5,x=1.23,y=5-1.23 (value after rounded). I think it is just a "lucky" ...
