This example shows how to plug a third-party serialization library into the
Durable Task Python SDK by implementing a custom
DataConverter. It uses
pydantic to serialize and validate payloads, and
proves the integration works end-to-end with in-process tests that need no
sidecar, emulator, or Azure resources.
You can copy this entire folder into a new directory and run it as a standalone project.
By default the SDK serializes payloads with its built-in JSON codec
(JsonDataConverter), which understands builtins, dataclasses, and objects that
expose to_json() / from_json() hooks. It does not know about pydantic
models. Supplying a custom DataConverter lets you route serialization through
any library you like — pydantic, attrs, marshmallow, a schema registry, an
encryption layer, etc. — at every payload boundary the SDK touches.
Both the worker and the client accept a data_converter argument, and the SDK
routes every payload boundary through it — orchestrator / activity / entity
inputs and outputs, external events, and custom status:
converter = PydanticDataConverter()
worker = DurableTaskSchedulerWorker(..., data_converter=converter)
client = DurableTaskSchedulerClient(..., data_converter=converter)Important
Pass an equivalent converter to both the worker and the client. A payload serialized by one side is reconstructed by the other, so they must agree on the format.
A DataConverter implements three methods:
| Method | Direction | Used when |
|---|---|---|
serialize(value) |
Python value → JSON string | Any value leaves the process |
deserialize(data, target_type) |
JSON string → Python value (optionally typed) | A value arrives and the SDK knows the target type (from a function annotation, return_type=, or a typed client accessor) |
coerce(value, target_type) |
already-parsed value → typed value | The SDK already holds a parsed value (e.g. entity state) |
The converter in src/converter.py recognizes
pydantic.BaseModel subclasses and uses pydantic for them, delegating
everything else to the default JsonDataConverter. This "handle my types,
delegate the rest" shape is a good starting point for a real converter — it
costs nothing for non-pydantic payloads.
Note
To stay focused on the seam, this example only intercepts when a model is the
top-level type. A model nested inside another model (like Order.items)
round-trips because pydantic recurses on its own, but a model nested in a
top-level generic the SDK rebuilds directly — e.g. return_type=list[OrderItem]
or an input annotated dict[str, OrderItem] — is not intercepted and falls to
the default codec, which leaves the elements as raw dicts. A production
converter would recurse into such generics (for example via
pydantic.TypeAdapter).
There is one extra detail for reconstructing inbound orchestrator/activity
inputs. Before the SDK hands an input to your converter, it asks the converter
whether the function's annotated input type is something it can rebuild, via
DataConverter.can_reconstruct(target_type). The default implementation
recognizes dataclasses and from_json()-capable types (and Optional / list
wrappers) — it does not know about pydantic models, so without an override an
input annotated order: Order would arrive as a plain dict.
The converter overrides can_reconstruct to also recognize
pydantic.BaseModel subclasses, deferring everything else to the same
JsonDataConverter fallback it uses for serialization:
def can_reconstruct(self, target_type):
if _is_model_type(target_type):
return True
return self._fallback.can_reconstruct(target_type) # dataclasses, from_json, ...The base DataConverter.can_reconstruct is conservative — it returns False,
so a converter only claims the types it actually rebuilds. Outbound values,
return_type= arguments, and typed client accessors (state.get_output(Receipt))
don't depend on this hook — they pass the type to the converter directly.
custom_data_converter/
├── README.md
├── requirements.txt
├── src/
│ ├── __init__.py
│ ├── converter.py # PydanticDataConverter — the integration point
│ ├── workflows.py # pydantic models + orchestrator/activities
│ └── app.py # runs against a real DTS backend / emulator
└── test/
├── __init__.py
└── test_custom_converter.py # in-process proof using the in-memory backend
The models in src/workflows.py are plain
pydantic.BaseModel subclasses — not dataclasses — so they only round-trip
correctly because of the custom converter. The tests in
test/test_custom_converter.py verify:
- A pydantic
Orderpassed as orchestration input arrives at the orchestrator/activity as a validated model instance (attribute access), not a raw dict. - The orchestration's pydantic
Receiptresult is reconstructed, typed, on the client viastate.get_output(Receipt). - The wire payload is genuine pydantic JSON (
model_dump_json), confirming the custom converter — not the default codec — handled it. - An input that violates a pydantic constraint fails the orchestration with a validation error, instead of passing bad data through.
- For contrast, the default
JsonDataConvertercannot serialize a pydantic model at all (it raisesTypeError) — which is exactly what motivates the custom converter.
-
Copy this folder to a new location and
cdinto it:cd custom_data_converter -
Create and activate a virtual environment:
Bash:
python -m venv .venv source .venv/bin/activatePowerShell:
python -m venv .venv .\.venv\Scripts\Activate.ps1 -
Install dependencies:
pip install -r requirements.txt
From the custom_data_converter/ directory:
pytest test/This is the self-contained proof: it runs the full orchestration in-process against the in-memory backend.
Start the DTS emulator, then from the
custom_data_converter/ directory:
python -m src.app