7,631 questions
3
votes
1
answer
107
views
Extracting decimal and negative numbers from PDF, where decimals look like exponents in the PDF
I am processing a set of PDF files using Python.
From each PDF I need to extract several monetary amounts and then write them to an Excel file.
The problem is that in the PDF the decimals are written ...
0
votes
2
answers
77
views
How to auto-adjust the width of an input field based on character length of the decimal displayed within?
I'm working on a very simple program that takes a user-specified number (can be an integer or a float of any length) and performs one specific multiplication operation on that number. The program then ...
- 1
3
votes
0
answers
85
views
How can I make a fields.Monetary keep 5 decimal places?
I need a monetary field to keep 5 decimal places (e.g. 12.34567), but Odoo version 18.0 enterprise keeps rounding to the currency’s precision (2 decimals for my company currency).
Here is my code:
...
- 31
3
votes
2
answers
64
views
Why does decimal.Context.create_decimal_from_float() not round near-zero values?
Contrary to what the docs might suggest, create_decimal_from_float() does not always round to its specified precision. As the last line of the following example shows, when a near-zero value is passed ...
- 53
0
votes
4
answers
289
views
Rounding floats / decimals in e-commerce (PHP 8.3 and Laravel 11)
So due to the fact that product quantities are not limited to integers but can store decimals up to 4 digits, I can have a product with quantity of 69.4160 square meters of something.
Now assuming the ...
- 5,472
1
vote
2
answers
96
views
Calculating VAT & Gross amounts from an onchange event
I have a php input form that I enter a 'Net Price' into mysql.
I have a seperate page that pulls that data into a table in a 'Quotations' page via echo.
I am trying to automatically calculate VAT and ...
0
votes
0
answers
107
views
In C#, why are 0.025f and 0.025 equal to 0.025m? [duplicate]
In C#, I have this code:
float initialResolutionFloat = 0.025f;
double initialResolutionDouble = 0.025;
decimal initialResolutionDecimal = 0.025m;
decimal resolutionFloat = (decimal)...
- 533
0
votes
0
answers
64
views
How to filter out rows that is less than 5 decimal places on 4 float 64 columns?
I am trying to clean a dataset for a case study, here is a problem that I have encountered:
I am trying filter out rows that is less than 5 decimal places (leaving .12345 or .12346 etc.) while ...
2
votes
1
answer
107
views
Change the number of decimals in stat_compare_means [duplicate]
I know there are some questions similar to this but I have tried the suggested answer and I was not able to figure out how to change the number of decimals in stat_compare_means
Maybe it is because I ...
3
votes
2
answers
149
views
Finding the Least Common Multiplier to Turn a Vector of Numbers Into Integers
I have the following vector.
Vector_1 <- c(0.1, 0.9, 0.5, (1 / 3), 0.5344)
I want to multiply each element of this vector by the same number such that each number in the resulting vector is an ...
- 1,026
-3
votes
1
answer
89
views
Can't eliminate floating point errors with Decimal [duplicate]
I'm using Decimal everywhere to try and get rid of floating point errors, but it's persisting.
Here's my test code
from decimal import Decimal
Demands = [9,12,8,11]
fM = Decimal(10)
alpha = Decimal(....
- 503
0
votes
0
answers
125
views
string format a Decimal instance using an underscore as a thousands separator?
I can easily do this for floats using an f-string like this:
x = float(123456789.123456)
print(f"{x:_f}")
prints 123_456_789.123456 exactly as expected.
But if try the same technique for a ...
- 951
0
votes
0
answers
69
views
Decimal item quantity in WooCommerce [duplicate]
// Add custom quantity setting fields to the admin product pages
add_action('woocommerce_product_options_pricing', 'wc_qty_add_product_field');
function wc_qty_add_product_field() {
global $...
0
votes
1
answer
74
views
Unexpected Decimal Separator Behavior with CultureInfo in .NET
In the attached images, you can observe that I am explicitly using both InvariantCulture and a new instance of CultureInfo("en-US") when converting a Double to a String:
value.ToString(...
- 484
-1
votes
1
answer
95
views
How can decimal values be totalled in a Windows batch file for loop [closed]
I have been trying to read a comma delimited csv file and total a value from each record using the code below that I have put together from various searches:
@echo off
setlocal EnableDelayedExpansion
...
- 47
1
vote
2
answers
180
views
How can I convert a large decimal string to an equivalent hex string?
I am trying to implement a custom 1024 bit integer datatype in C. I want to implement operations like addition, subtraction and multiplication. Here is the structure I am defining:
typedef enum{...
- 137
0
votes
1
answer
350
views
Handling Large Decimal Numbers (200+ digits/decimals) in TypeScript: String vs Number and parseFloat() Conversion
I'm working with very large decimal numbers in TypeScript that can have up to 200 decimal places. Currently, I'm storing these numbers as strings to preserve precision.
const largeDecimal: string = &...
- 59
0
votes
1
answer
91
views
Unable to Change Comma to a Dot in Decimals in R [closed]
I have a .csv file that I’m having trouble processing because the numbers use commas for decimals, which causes issues when reading the file in R. Excel interprets commas for decimals, while R uses ...
- 59
0
votes
1
answer
54
views
In .NET will a `decimal` equal to an `int` always cast to that `int`
In .NET, assuming that a decimal is not so large that it cannot be represented (or cause an overflow), will a decimal that is equal to an int always cast to the int that it is equal to ?
Same question ...
- 11.5k
1
vote
2
answers
89
views
Convert float base 2 to base 10 to 100+ decimal places [closed]
I'm trying to find more decimal places to the 'Prime Constant'. The output maxes out at 51 decimal places after using getcontext().prec=100
from decimal import *
getcontext().prec = 100
base = 2
s = &...
- 23
0
votes
2
answers
87
views
SQL separate units and decimals from price
I have price column in MySQL contains this values :
price
1020.2200
202.0000
0.5000
820.6700
I want to split at the floating point.
Here is what I'm trying to get :
price
p_units
p_decimals
1020.2200
...
- 301
-1
votes
2
answers
299
views
VBA - Get Fractional Part of a Number, problems with Int() [duplicate]
I am trying to obtain the fractional part of a number in VBA in Excel. Is there a particular method that works best to get the fractional part of a number?
I have tried two different methods. One of ...
- 3
-1
votes
1
answer
78
views
In Python, can you get the full utility of a long decimal while only displaying two decimal places?
I have a pair of boxes. One is for labor hours. The other is for labor dollars. When you enter a labor hour amount, it automatically calculates and fills the labor dollar box. When you enter a labor ...
- 147
0
votes
1
answer
83
views
Way to handle both "," and "." as decimal separators
I have a program where the user can copy an excel table and paste it on the program itself for data analysis. The only value allowed for the table is float. The problem is that there are 4 different ...
- 3
2
votes
2
answers
284
views
How to turn decimal to byte array but it has to be lexicographically sortable?
As for double BitConverter.GetBytes(double) reversed returns array that if lexicographically sorted will be sorted the same as (positive) doubles. Can the same be done for decimal?
For example:
var ...
- 3,991
3
votes
2
answers
346
views
df.to_excel converts Decimal columns to text
I have a DataFrame where some columns use the Decimal datatype. I'd like to export this data to an Excel spreadsheet, keeping the decimal/numeric format. However, my code is converting the columns to ...
- 137
-1
votes
1
answer
108
views
Why is my test not losing precision by using floats for percentages, versus whole-integer basis points, for interest in Rust? [duplicate]
This is absolutely not a duplicate of Why not use Double or Float to represent currency? - I have edited the question further to make this even more explicit.
Why is my test not losing precision by ...
- 126k
1
vote
2
answers
76
views
PHP create a non-repeating range of float numbers using php random generator functions
I am managing a function that can vary a range_pool of numbers with set decimals between them. I would like to find a solution that ensures that these numbers never repeat (create duplicates) when ...
- 299
1
vote
0
answers
52
views
Inconsistent behavior of float datatype in Pandas dataframe for high precision decimal values
Pandas dataframe gives incorrect decimal values with high precisions.
For example:
The data in oracle tables are
45.200000000000003
12.699999999999999
99.745663
The data type in oracle is number(28,15)...
- 41
4
votes
1
answer
6k
views
Incrementing to the last decimal in python
I want to write function in python when given float or string for example 0.003214 to return me 0.003215, but if I give it 0.00321 to return 0.003211, it will apply on other floats also like 0.0000324 ...
0
votes
1
answer
85
views
Calculate the correct amount after doing a Discount and Contract weight
I have this code inside a .net core console application:-
var currentDate = startdate;
while (currentDate <= enddate)
{
// Get the days in the current month
var daysInMonth = DateTime....
1
vote
1
answer
138
views
I cant send Decimal variable properly (with '.' or ',')
Here is my DiscountRate variable in my model.
public decimal DiscountRate { get; set; }
Here is my input in view.
@Html.TextBoxFor(x=>x.DiscountRate, new { @class = "form-control", @...
- 31
0
votes
2
answers
75
views
How do I convert string representation of duration into decimal format in R?
In my data file, I have a column that represents the total duration that individuals students spend on a website working through different learning modules. For example, a student spent 11 hours and ...
0
votes
1
answer
80
views
Correct Decimal to Floating-Point using Big Integers in Ruby
I'm trying to implement, for learning purposes, the core part of an algorithm that converts decimal strings to 64 bit floating point numbers.
I'm using the explanation from this page as a guide: https:...
- 1,017
0
votes
0
answers
79
views
Difference in handling zero after decimal between web and android
I noticed Flutter has different ways of handling zero after decimals between the web and Android.
for example:
Double Z = 0;
Double Y = 0;
Double X = 0;
Double W = 0;
Z = Y + X * (4 - W));
Print (Z);...
- 13
0
votes
0
answers
62
views
decimal.InvalidOperation in Tortoise ORM
Here's how my model is defined:
class User(Model):
id = BigIntField(primary_key=True, source_field="user_id")
text = CharField(max_length=255, null=True)
decimal = DecimalField(...
- 147
5
votes
3
answers
237
views
Polars DataFrame - Decimal Precision doubles on mul with Integer
I have a Polars (v1.5.0) dataframe with 4 columns as shown in example below. When I multiply decimal columns with an integer column, the scale of the resultant decimal column doubles.
from decimal ...
- 313
6
votes
1
answer
107
views
Decimal shift behaves bizarrely if constructed from float
Consider the following Python code:
from decimal import Decimal
d = Decimal("1.23")
print(f"{d = }, {d.shift(1) = }")
When I execute it in Python 3.12.4, I get this output:
d = ...
- 951
0
votes
1
answer
56
views
Clarification on "each digit represents an increasing power of 10" in number representation
I'm trying to understand a statement I came across: "each digit represents an increasing power of 10." This seems unclear to me because, from my understanding, each digit itself does not ...
-1
votes
1
answer
112
views
round function is inconsistent in rounding 2 decimal places [closed]
The code below is taken from my program. This block formats these float64 columns to 2 decimal places but rows 24-26 really won't change.
Here's a minimal reproductive example (I hope this is ...
0
votes
0
answers
55
views
Is there a way to get decimal accuracy to 19 spots post decimal? [duplicate]
I have the following fields:
Field1:
0.00
Field2:
94777.58
Field3:
223000.00
FinalNumber = (Field1 + Field2)/Field3
All fields are Decimal(11,2).
Trying to get this FinalNumber value to 19 spots ...
- 431
1
vote
1
answer
83
views
C++ decimal separator syntax mistake
I recently made a syntax mistake by writing this:
b = (float)a*0,1;
instead of:
b = (float)a*0.1;
I am surprised to find out that the first line don't even put a compilation error. I am even more ...
- 23
1
vote
1
answer
109
views
Use 2 decimals for discount amount in Woocommerce with 0 decimals settings
I am using 0 decimals under wc-settings&tab=general which is working fine for my usecase. But now adding a discount for a specific payment method which is set to 10 percent at the moment.
For an ...
- 89
2
votes
0
answers
134
views
iOS numeric keyboard doesn't have a period and the decimal inputmode is not considering properly the comma
I'm having a problem on iOS devices when filling a <input type="number" inputmode="decimal">. The iOS keyboard for these inputs shows numbers and a comma, only.
iOS keyboard
...
0
votes
1
answer
57
views
How to reduce decimal places to just 2?
I have this query in BigQuery
SELECT Id,
AVG(VeryActiveMinutes) AS avg_very_active,
SUM(VeryActiveMinutes) AS total_very_active,
SUM(SedentaryMinutes) AS total_sedentary
FROM `...
1
vote
0
answers
90
views
Is Decimal a lot slower in MacOS Swift than Decimal in Python?
[update]fyi: I did try running the same test below with GMP lib in Swift and it is about 10x faster than Swift Decimal which I imagine the upper bound would be. Still confused why Python Decimal would ...
- 135
2
votes
1
answer
85
views
Return generic to its discrete type
I run a bunch of calculations using Decimal and Double. Without generics there'd be a lot of repetitive code. There are places I eventually need the Decimal in a more native format but I can't seem to ...
- 135
0
votes
1
answer
1k
views
How to convert Hex to Dec in Codesys ST?
I'm using Codesys V3 ST language
I'm writing a code to communicate motor controller unit which uses CANOpen protocol
I want to use CAN raw messages to communicate
I put CAN bus messages bytes in an ...
- 35
-3
votes
1
answer
124
views
Converting a HEX color value to RGB in JavaScript
I have this code:
let hexInt = parseInt(hex, 16);
let r = (hexInt >> 16) & 255;
let g = (hexInt >> 8) & 255;
let b = hexInt & 255;
I know that a HEX digit requires 4 bits to ...
- 1,613
0
votes
1
answer
124
views
How to make python use more than 15 decimals in calculations?
In this question I made a claim that this construction of pentagons will always converge.
I drew the first 50 iteration using Geogebra here. However, zooming in on the 50th iteration resulted in ...
- 127