Databricks pyspark pandas error with numpy

Question

I am getting the following error when using pyspark pandas:

PandasNotImplementedError: The method pd.Series.__iter__() is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead

Here is my code:

import pyspark.pandas as ps

df_mas=(spark.read.format("csv").option ("header", "true"). load (driver.config["OutputFiles"])
df=df_mas.pandas_api()
df["MAUS"] = nр.where(df.MAUS=="NHTT"),"MHINC", df.MAUS)
display (df)

Dinesh Kumar Rajendran · Accepted Answer · 2024-05-17 05:27:57Z

0

Please try upgrading your spark version in cluster or try downgrading the pandas version while by installing a lower version like below in your code before importing pandas (import pandas as ps)

%pip install -U pandas==1.5.3

answered May 17, 2024 at 5:27

Dinesh Kumar Rajendran

73 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Databricks pyspark pandas error with numpy

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related