(ffi)= (extension_guide)=
This section is for people writing a library that plugs into datafusion-python — a package that contributes table providers, functions, a catalog, extension codecs, or a query planner, usually written in Rust and exposed through PyO3.
Two neighbouring audiences are served elsewhere:
- Using an extension library someone else published:
{ref}
user_guide_extensions. - Changing datafusion-python itself, including the framing that makes this
protocol work: {doc}
../contributor-guide/index.
The protocol described here is a public, versioned contract. When a hook's
signature changes, the change is documented in
{doc}../user-guide/upgrade-guides with a before-and-after, and objects built
against a different DataFusion major version are rejected with a clear error
rather than used as-is.
If you have not built an extension before, read {doc}why-ffi and
{doc}capsule-protocol in order. Together they explain why your library must
not depend on the datafusion-python crate, and the one convention every hook
follows. After that the pages are independent — go to the one matching what
you are contributing.
A single query can involve three independent native libraries, and much of this section only makes sense once they are distinct in your head:
- datafusion-python — the host. Owns the session, and decodes whatever comes back from the other two.
- A provider library — owns table providers, catalogs, and functions, plus the codecs that serialize them.
- A planner library — owns a query planner and the configuration it needs.
The worked examples in this repository use two separate crates,
datafusion-ffi-example and datafusion-ffi-query-planner-example, so
each role has a distinct shared-library identity. A real library may play more
than one role; keeping them separate in the examples is what makes the
boundaries observable.
The session owns the codecs used for the exchange and supplies them to the foreign planner. That is what lets the planner decode provider-owned objects, and lets datafusion-python decode the physical plan the planner returns.
Every integration point is a dunder method named __datafusion_*__. The
convention is uniform enough to be worth stating once: your object exposes the
getter, datafusion-python calls it, and it returns a PyCapsule wrapping an
FFI-safe struct. See {doc}capsule-protocol for what that means and
{ref}extension_getter_argument for the argument every getter in the middle
group receives.
Each hook's capsule name follows from its own name by
{ref}the naming rule <extension_capsule_protocol>, so it is not repeated
here.
| Hook | Contributes | Argument | Documented on |
|---|---|---|---|
__datafusion_table_provider__ |
one table | codec source | {doc}table-providers |
__datafusion_table_provider_factory__ |
tables built by CREATE EXTERNAL TABLE |
codec source | {doc}table-providers |
__datafusion_catalog_provider__ |
a named set of schemas | codec source | {doc}table-providers |
__datafusion_catalog_provider_list__ |
the whole catalog namespace | codec source | {doc}table-providers |
__datafusion_schema_provider__ |
a named set of tables | codec source | {doc}table-providers |
__datafusion_table_function__ |
a table-valued function | session | {doc}functions |
__datafusion_scalar_udf__ |
a scalar function | none | {doc}functions |
__datafusion_aggregate_udf__ |
an aggregate function | none | {doc}functions |
__datafusion_window_udf__ |
a window function | none | {doc}functions |
__datafusion_logical_extension_codec__ |
a logical codec | session | {doc}codecs |
__datafusion_physical_extension_codec__ |
a physical codec | session | {doc}codecs |
__datafusion_codec_id__ |
the wire id a codec's payloads carry | — | {ref}extension_codec_ids |
__datafusion_query_planner__ |
a query planner | session | {doc}query-planners |
__datafusion_session_components__ |
a bundle's codecs | ctx |
{doc}bundles |
__datafusion_session_planner__ |
a bundle's planner, wrapping fallback |
ctx, fallback |
{doc}bundles |
__datafusion_physical_optimizer_rule__ |
a physical optimizer rule | none | {ref}extension_other_hooks |
__datafusion_extension_options__ |
typed entries in the session config | none | {ref}extension_other_hooks |
__datafusion_task_context_provider__ |
the host's task context | none | {ref}extension_task_context_provider |
Three rows are not like the others. __datafusion_task_context_provider__ is
implemented by the host, not by your library — you read it off the session
you are handed. __datafusion_codec_id__ is a plain string attribute rather
than a method returning a capsule. And the two session_ hooks are dispatched
from Python and return objects rather than capsules.
"codec source" in the argument column means the value is something you can
read the host's logical extension codec off, which is not always a session.
{ref}extension_getter_argument explains why, and what to do with it.
python/tests/test_docstrings.py compares this table against the hook names
the package actually dispatches, so a hook added or renamed without touching
this page fails the suite.
:maxdepth: 2
why-ffi
capsule-protocol
table-providers
functions
codecs
bundles
query-planners
other-components
sessions
checklist