Repository navigation
Inconsistent behavior serializing python numbers and decimals #826
Description
Description
Hello all - we use msgspec to encode json of api call responses.
We recently discovered that there is a discrepancy between how python floats and decimals are encoded.
In particular, NaN and Infinity float values are encoded as "null", but NaN and Infinity Decimal values are interpolated directly, which produces invalid json.
We prefer the behavior of encoding these invalid numeric values as null.
I couldn't find a way to work around this without preprocessing our responses before passing them to msgspec - it doesn't seem like there is a way to add a custom hook for the encoding of known types like decimal. However, it is possible that I missed a way to do so.
Can you all please advise on whether this behavior is expected or not?
Replication code:
import msgspec
import math
import decimal
msgspec_encoder = msgspec.json.Encoder(decimal_format="number")
def encode(arg: object) -> object:
return msgspec_encoder.encode(arg).decode()
def test(arg: object) -> None:
wrapped_arg = {"value": arg}
print(f"`{wrapped_arg}` encodes as `{encode(wrapped_arg)}`")
print(f"Version: {msgspec.__version__}")
test(1)
test(math.inf)
test(math.nan)
test(decimal.Decimal(1))
test(decimal.Decimal("Infinity"))
test(decimal.Decimal("NaN"))
Output:
Version: 0.19.0
`{'value': 1}` encodes as `{"value":1}`
`{'value': inf}` encodes as `{"value":null}`
`{'value': nan}` encodes as `{"value":null}`
`{'value': Decimal('1')}` encodes as `{"value":1}`
`{'value': Decimal('Infinity')}` encodes as `{"value":Infinity}`
`{'value': Decimal('NaN')}` encodes as `{"value":NaN}`
Activity
provinzkraut commentedon Mar 20, 2025
Not saying this is invalid, but it's documented behaviour at least:
https://jcristharif.com/msgspec/supported-types.html#decimal
https://jcristharif.com/msgspec/supported-types.html#float
ucwillg commentedon Mar 20, 2025
That's a fair. The next section of the docs on decimals discusses the mode we're using here and doesn't have a note similar to the one you cited for floats.
In any case, the default behavior of either encoding mode is less important to us than having some way to efficiently encode all valid decimal.Decimal values as valid JSON, including NaN and Infinity values. We're happy to configure something custom, but it's not clear to me if there's a way to achieve this behavior without preprocessing the entire data structure before handing it off to msgspec.