I've just had a look at pregel.py, and there are a few things I'm not entirely convinced about (I'm aware most of them have nothing to do with your issues):

Excessive use of Any: Since this is a Python wrapper for the class Pregel.scala, and the corresponding Scala methods use specific types like Column, it would be more helpful (and safer) to use Column in the Python type hints as well, instead of Any.
Redundant type conversions: There are several places where values are being converted to types they already are. These conversions seem unnecessary:

    def setMaxIter(self, value: int) -> "Pregel":
        self._java_obj.setMaxIter(int(value))
        return self

    def setCheckpointInterval(self, value: int) -> "Pregel":
        self._java_obj.setCheckpointInterval(int(value))
        return self

    def setEarlyStopping(self, value: bool) -> "Pregel":
        self._java_obj.setEarlyStopping(bool(value))
        return self

GraphFrames connect: I don't much about how Spark Connect works (yet), but ...

Why are we duplicating code from pregel.py in graphframe_client.py? Wouldn't it be better to reuse the functionality from pregel.py instead?

 @staticmethod
 def msg() -> Column:
     return F.col("_pregel_msg_")

 @staticmethod
 def src(colName: str) -> Column:
     return F.col("src." + colName)

 @staticmethod
 def dst(colName: str) -> Column:
     return F.col("dst." + colName)

 @staticmethod
 def edge(colName: str) -> Column:
     return F.col("edge." + colName)

Why the PregelConnect class is located in the same graphframe_client.py and this file has more than 800 lines of code? Can't we split the classes into separate files for better maintainability?

feat: sync python with scala-core #586

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions