-
Notifications
You must be signed in to change notification settings - Fork 108
Python API todos #1776
Copy link
Copy link
Closed
Labels
python-sdkIssues related to the feldera python sdkIssues related to the feldera python sdk
Description
- Publish docs on the website
- Add a
foreach_chunkmethod for streaming HTTP output - py: implement foreach_chunk method for streaming HTTP output #1792 - Pass the new
backpressureflag to the output HTTP connector to prevent it from dropping chunks. Alternatively, make this flag configurable, but the default for the Python API should be true. - py: apply backpressure by default for http egress #1791 - Kafka connector support - py: kafka connector #1807
- HTTP get connector support (docs) - py: HTTP GET input connector #1816
- Run Python SDK tests in ci. - ci: run python tests #1873
- The input HTTP connector should split large inputs into multiple HTTP requests. py: properly serialize DataFrames with Timestamp columns #1846
- Pipeline state management / tracking
- Allow feeding new input data frames at runtime py: implement
SQLContext.wait_for_completion#1872 -
wait_for_completion- wait for a running pipeline to complete py: implementSQLContext.wait_for_completion#1872 - [SQL] Support LOCAL views, which do not produce circuit outputs #1851 introduced local views. We need to add support for them in the Python SDK, probably in the form of a
register_local_viewmethod. I would go a step farther and renameregister_viewtoregister_output_view, so that users must make an explicit choice to expose the view externally. - Improve auto-generated docs structure. Is it possible to flatten the docs so that the main page includes the intro section and
felderapackage docs inlined? Thefeldera.restsubpackage on the other hand could live in a separate page so that it doesn't get in the way of browsing the frontend API? - Improve py docs #1942
Post-API-refactoring tasks:
- Interact with the backend to maintain the state of the program: rather than tracking tables and views locally, the Python SDK should forward all register_view/register_table requests to the backend and get back validation results and updated program schema. In addition to providing instant feedback about invalid SQL, this will allow the API to accept arbitrary chunks of SQL code (e.g., multiple table/view declarations). This requires changes to the backend.
- Delta input/output connectors: implement builder pattern similar to other connectors, document connector config with examples.
- Output buffer config. Once we switch to using the builder pattern for all connectors, add methods to configure the output buffer for all of them.
- Support for schema inference from Delta tables
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
python-sdkIssues related to the feldera python sdkIssues related to the feldera python sdk