Is there an efficient way to format Decimal?

Question

It is nice, easy and fast to use format-strings in python. So I never considered the performance penalty of this operation. Some time ago I switched my program from float data type to Decimal to eliminate rounding errors. Some performance degradation was expected and it is ok. But I'm surprised to see how large performance penalty I have just from printing a log of formatted Decimal numbers.

Below I illustrate the difference with cProfile results. The question is - are there any efficient ways to have formatted Decimal numbers in python?

Here is a test for float-number formatting:

from cProfile import Profile

a_float = 1234567890.12345

def format_float(value: float) -> str:
    if value is None:
        return ''
    result = f"{value:+,.6f}"
    return result

def test_float():
    for i in range(1000000):
        b = format_float(a_float)

p = Profile()
p.runcall(test_float)
p.print_stats()

it gives result: 1000002 function calls in 0.839 seconds

And here is the same test with Decimal:

from decimal import Decimal
from cProfile import Profile

a_decimal = Decimal('1234567890.12345')

def format_decimal(value: Decimal) -> str:
    if value is None:
        return ''
    result = f"{value:+,.6f}"
    return result

def test_decimal():
    for i in range(1000000):
        b = format_decimal(a_decimal)

p = Profile()
p.runcall(test_decimal)
p.print_stats()

which results in: 55000002 function calls in 27.739 seconds

Is there any way to format Decimal nicely in shorter time?

The discussion in comments brought nice spot - that others don't have such a problem. Here I think the output of the profiler is of some interest.

For the case with float it is pretty short and ends with {method 'disable' of '_lsprof.Profiler' objects} that suggest some optimization I think:

>>> p.print_stats()
         1000002 function calls in 0.735 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1000000    0.564    0.000    0.564    0.000 <stdin>:1(format_float)
        1    0.171    0.171    0.735    0.735 <stdin>:1(test_float)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

while with Decimal there are a lot of lines:

>>> p.print_stats()
         47000029 function calls in 23.731 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1000000    0.625    0.000   23.395    0.000 <stdin>:1(format_decimal)
        1    0.335    0.335   23.731   23.731 <stdin>:1(test_decimal)
  1000000    0.947    0.000    1.795    0.000 _pydecimal.py:2622(_rescale)
  1000000    3.218    0.000   22.771    0.000 _pydecimal.py:3758(__format__)
  1000000    0.533    0.000    0.665    0.000 _pydecimal.py:3844(_dec_from_triple)
        1    0.000    0.000    0.000    0.000 _pydecimal.py:3902(__init__)
        5    0.000    0.000    0.000    0.000 _pydecimal.py:3938(_set_integer_check)
        2    0.000    0.000    0.000    0.000 _pydecimal.py:3952(_set_signal_dict)
        9    0.000    0.000    0.000    0.000 _pydecimal.py:3963(__setattr__)
  1000000    0.294    0.000    0.405    0.000 _pydecimal.py:448(getcontext)
  1000000    1.974    0.000    3.767    0.000 _pydecimal.py:6188(_parse_format_specifier)
  1000000    0.870    0.000    0.988    0.000 _pydecimal.py:6268(_format_align)
  1000000    1.734    0.000    1.822    0.000 _pydecimal.py:6295(_group_lengths)
  1000000    6.271    0.000   10.549    0.000 _pydecimal.py:6318(_insert_thousands_sep)
  1000000    0.236    0.000    0.236    0.000 _pydecimal.py:6355(_format_sign)
  1000000    1.390    0.000   13.162    0.000 _pydecimal.py:6365(_format_number)
  3000000    0.466    0.000    0.466    0.000 _pydecimal.py:820(__bool__)
  1000000    0.132    0.000    0.132    0.000 {built-in method __new__ of type object at 0x7e4f98d56d40}
        7    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
 16000000    0.979    0.000    0.979    0.000 {built-in method builtins.len}
  4000000    0.732    0.000    0.732    0.000 {built-in method builtins.max}
  4000000    0.524    0.000    0.524    0.000 {built-in method builtins.min}
        1    0.000    0.000    0.000    0.000 {built-in method fromkeys}
  4000000    0.277    0.000    0.277    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'copy' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
  1000000    0.112    0.000    0.112    0.000 {method 'get' of '_contextvars.ContextVar' objects}
  1000000    0.613    0.000    0.613    0.000 {method 'groupdict' of 're.Match' objects}
  1000000    0.289    0.000    0.289    0.000 {method 'join' of 'str' objects}
  1000000    1.180    0.000    1.180    0.000 {method 'match' of 're.Pattern' objects}
        1    0.000    0.000    0.000    0.000 {method 'set' of '_contextvars.ContextVar' objects}

In fast test ran by Suramuthu R here here I see that his Decimal test is similar to my float and ends with the line:

{method 'disable' of '_lsprof.Profiler' objects}

It looks like some optimization is switched off for Decimal on my PC. Does anyone have idea how to check it?

Where/how are you running these tests? I'm getting 1000002 function calls in 1.049 seconds for floats and 1000002 function calls in 1.126 seconds for Decimal on Python 3.8 — DeepSpace
– DeepSpace, Commented Feb 11, 2024 at 9:58
Hi DeepSpace. Initially I ran in PyCharm and Jupyter just due to convenient GUI. After your comment I ran in console just like command "python test.py" - it gave me sligthly better result (19 seconds VS 0.7 seconds) but still quite slow. I run Arch Linux with Python 3.11.6. Maybe it is about how Decimal is installed?.. but I don't see any specific packet for it... Which OS do you use with such great results? — StarterKit
– StarterKit, Commented Feb 11, 2024 at 10:30
In python 3.10.12, this is the result. for float : 1000002 function calls in 0.637 seconds; for decimal :1000002 function calls in 0.419 seconds; Screenshot. In online-compiler, it takes more time. Even then, for decimal, it takes lesser time. My suggestion is to get your python version updated to 3 and above and to test the result in terminal. — Suramuthu R
– Suramuthu R, Commented Feb 11, 2024 at 10:31
Ok, guys, Thank you, you give me hope... But... may you give more details about your setups? It appears I need to dig dipper and do more experiments but I need some direction. (I'll try with lower python version later today) — StarterKit
– StarterKit, Commented Feb 11, 2024 at 10:34
You are right, the issue is that your setup is using pure-python implementation of decimal for some reason (you can see _pydecimal.py in the statistics) whereas it should be using C implementation. See here. — yuri kilochek
– yuri kilochek, Commented Feb 11, 2024 at 10:59

FJSevilla · Accepted Answer · 2024-02-11 12:04:36Z

2

This happens in the version of Python provided by the Arch repositories because it forces the use of several libraries installed on the system when compiling Python (see Python PKGBUILD), among them libmpdec.

So, the cause is the lack of the optional mpdecimal dependency, whose absence forces Python to use the Python implementation of Decimal, as yuri kilochek correctly pointed out in the comments:

... the problem is that your configuration is using a pure python decimal implementation for some reason (you can see _pydecimal.py in the stats) while it should be using a C implementation.

Installing the mpdecimal package fixes the problem:

sudo pacman -S install mpdecimal

Being faster than for the float case, 1000002 function calls in 1.104 seconds in this same system.

edited Feb 11, 2024 at 12:04

answered Feb 11, 2024 at 11:43

FJSevilla

4,5431 gold badge16 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

StarterKit Over a year ago

Great! You are right, mpdecimal was not installed and it solved my problem in a moment. Thanks a lot to all who contributed!

jackal · Accepted Answer · 2024-02-11 12:16:31Z

0

The behaviour appears to differ significantly depending on platform.

For example, on MacOS 14.3.1 and Python 3.12.1 on Apple Silicon M2 this code:

from cProfile import Profile
from decimal import Decimal


def format_float(value) -> str:
    return f"{value:+,.6f}"

def test_value():
    global a_x
    for _ in range(1_000_000):
        format_float(a_x)

v = 1234567890.12345

for a_x in Decimal(v), v:
    p = Profile()
    p.runcall(test_value)
    p.print_stats()

...produces this output...

         1000002 function calls in 0.304 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.075    0.075    0.304    0.304 Feb11.py:10(test_value)
  1000000    0.229    0.000    0.229    0.000 Feb11.py:7(format_float)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}


         1000002 function calls in 0.449 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.077    0.077    0.449    0.449 Feb11.py:10(test_value)
  1000000    0.372    0.000    0.372    0.000 Feb11.py:7(format_float)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

...where we see far fewer calls being made than shown in the OP. Why formatting of Decimal is (apparently) faster than float is a mystery to me

answered Feb 11, 2024 at 12:16

jackal

29.1k3 gold badges10 silver badges28 bronze badges

1 Comment

StarterKit Over a year ago

Yes, as FJSevilla correctly found - my platform missed essential C-implementation package. I think it is logical that Decimal is faster. I mean float is a pure binary format while Decimal is stored in some more human-readable format to keep all these digits as I understand. So there seems to be less binary to human converstion steps involved.

Collectives™ on Stack Overflow

Is there an efficient way to format Decimal?

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related