py: kafka connector by abhizer · Pull Request #1807 · feldera/feldera

abhizer · 2024-05-30T10:53:51Z

also rename Client to FelderaClient

Is this a user-visible change (yes/no): yes

In a follow up PR, add builders for Connectors.

snkas

Looks good!

python/feldera/sql_context.py

snkas · 2024-05-30T14:46:00Z

python/tests/test_wireframes.py

+            "auto.offset.reset": "earliest",
+        }
+
+        kafka_format = JSONFormat().with_update_format(UpdateFormat.InsertDelete).with_array(False)


It could also be a class with default values for its instantiation JSONFormat(update_format=UpdateFormat.Raw, array=False), and then the user supplies arguments to change any of the defaults. Either works though.

I am split between having a default and requiring explicit configuration.
Having a default might mean that people just use the default and might have issues when the format doesn't match the data.

Better to not have defaults for this, given we don't have data on what the typical format might be.

lalithsuresh · 2024-05-30T14:49:59Z

This is a user-facing change and needs a changelog.

python/feldera/sql_context.py

mihaibudiu · 2024-05-30T15:51:24Z

python/feldera/sql_context.py

+        :param table_name: The name of the table.
+        :param connector_name: The unique name for this connector.
+        :param config: The configuration for the kafka connector.
+        :param fmt: The format of the data in the kafka topic.


can you provide a reference to the format descriptions?

Format takes an instance of JSONFormat or CSVFormat.

python/feldera/sql_context.py

mihaibudiu · 2024-05-30T15:52:08Z

python/feldera/sql_context.py

+        fmt = fmt.to_dict()
+
+        if fmt.get("config").get("update_format") is None:
+            raise ValueError("update_format not set in the format config try using method: .with_update_format()")


"; consider using"

mihaibudiu · 2024-05-30T15:53:03Z

python/feldera/sql_context.py

+        """
+        Tells Feldera to write the data to the specified kafka topic.
+
+        :param view_name: The name of the view whose output is sent to kafka topic.


it is really the view changes that are sent

mihaibudiu · 2024-05-30T15:54:10Z

python/tests/test_wireframes.py

+        ])
+        print("Topics ready")
+
+        # Produce of rows into the input topic


mihaibudiu · 2024-05-30T15:55:14Z

python/tests/test_wireframes.py

+        sql.connect_source_kafka(TABLE_NAME, "kafka_conn_in", source_config, kafka_format)
+        sql.connect_sink_kafka(VIEW_NAME, "kafka_conn_out", sink_config, kafka_format)
+
+        out = sql.listen(VIEW_NAME)


why so many empty lines?

mihaibudiu · 2024-05-30T15:55:54Z

python/feldera/formats.py

+
+    InsertDelete = 1
+    """
+    InsertDelete: When used, the data is inside a key either "insert" or "delete". This is used to represent changes.


I have trouble parsing this phrase

Me too. I am not sure how to word this properly.

mihaibudiu · 2024-05-30T20:17:13Z

python/feldera/formats.py

+
+    InsertDelete = 1
+    """
+    This is a way to represented changes in the data.


mihaibudiu · 2024-05-30T20:18:04Z

python/feldera/formats.py

+    InsertDelete = 1
+    """
+    This is a way to represented changes in the data.
+    When used, the actual JSON data is expected to be inside one of the keys "insert" or "delete".


I think that the example is clearer than this text, which can be removed.

just explain that 'id' and 'name' below are the columns of the table where the insertion and deletion happen.

mihaibudiu · 2024-05-30T20:18:43Z

python/feldera/formats.py

+
+    Raw = 2
+    """
+    This format represents an individual row in a SQL table or view.


It is really the same as an insertion, I presume

ryzhyk · 2024-05-31T17:04:38Z

python/docs/examples.rst

+To setup Kafka as the source use :meth:`.SQLContext.connect_source_kafka` and as the sink use
+:meth:`.SQLContext.connect_sink_kafka`.
+
+Both of these methods require a ``config`` which is a key-value pair.


Suggested change

Both of these methods require a ``config`` which is a key-value pair.

Both of these methods require a ``config`` which is a dictionary.

ryzhyk · 2024-05-31T17:06:09Z

python/docs/examples.rst

+        "auto.offset.reset": "earliest",
+    }
+
+    kafka_format = JSONFormat().with_update_format(UpdateFormat.InsertDelete).with_array(False)


Suggested change

kafka_format = JSONFormat().with_update_format(UpdateFormat.InsertDelete).with_array(False)

format = JSONFormat().with_update_format(UpdateFormat.InsertDelete).with_array(False)

ryzhyk · 2024-05-31T17:07:03Z

python/docs/examples.rst

+        "auto.offset.reset": "earliest",
+    }
+
+    kafka_format = JSONFormat().with_update_format(UpdateFormat.InsertDelete).with_array(False)


Mention that in addition to Kafka config, the Kafka connector also requires data format config and point to data format docs.

python/docs/examples.rst

python/feldera/formats.py

ryzhyk · 2024-05-31T17:13:37Z

python/feldera/formats.py

+            "array": False,
+        }
+
+    def with_update_format(self, update_format: UpdateFormat) -> Self:


Needs a comment

python/feldera/formats.py

python/feldera/sql_context.py

ryzhyk · 2024-05-31T22:02:33Z

Please also add links to Python docs from connector documentation pages (maybe it has to be a separate PR).

* also rename `Client` to `FelderaClient` Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> py: update the kafka test Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> py: update documentation * include an example for Kafka connector in the python docs * refactor `Client` to `FelderaClient` * add an entry in the changelog Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> py: update documentation Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com>

* py: HTTP GET input connector Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> * py: refactor: common function to check format to create connector Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> * py: update docs, rename `UpdateFormat` to `JSONUpdateFormat` Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> --------- Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com>

abhizer added the python-sdk Issues related to the feldera python sdk label May 30, 2024

abhizer requested a review from ryzhyk May 30, 2024 10:53

abhizer assigned ryzhyk and unassigned ryzhyk May 30, 2024

snkas approved these changes May 30, 2024

View reviewed changes

mihaibudiu reviewed May 30, 2024

View reviewed changes

abhizer added documentation Improvements or additions to documentation User-facing For PRs that lead to Feldera-user visible changes labels May 30, 2024

abhizer requested a review from mihaibudiu May 30, 2024 18:02

mihaibudiu reviewed May 30, 2024

View reviewed changes

abhizer mentioned this pull request May 31, 2024

Python API todos #1776

Closed

16 tasks

ryzhyk reviewed May 31, 2024

View reviewed changes

abhizer requested a review from ryzhyk June 3, 2024 15:22

abhizer mentioned this pull request Jun 3, 2024

add links to Python docs from connector documentation pages #1827

Open

ryzhyk approved these changes Jun 3, 2024

View reviewed changes

abhizer added 2 commits June 4, 2024 00:47

abhizer force-pushed the py_kafka branch from b1b31dd to 4d4ed53 Compare June 3, 2024 19:03

abhizer merged commit 4fee55f into main Jun 4, 2024

abhizer deleted the py_kafka branch June 4, 2024 02:35

	Both of these methods require a ``config`` which is a key-value pair.
	Both of these methods require a ``config`` which is a dictionary.

	kafka_format = JSONFormat().with_update_format(UpdateFormat.InsertDelete).with_array(False)
	format = JSONFormat().with_update_format(UpdateFormat.InsertDelete).with_array(False)

Conversation

abhizer commented May 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snkas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lalithsuresh commented May 30, 2024

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryzhyk commented May 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

abhizer commented May 30, 2024 •

edited

Loading

ryzhyk commented May 31, 2024 •

edited

Loading