Skip to content

fix: fix bug with na in the column labels in stack #659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gcf-merge-on-green merged 6 commits into from
May 6, 2024
Merged

fix: fix bug with na in the column labels in stack #659

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
032f700
fix: fix bug with na in the column labels in stack
milkshakeiii May 3, 2024
87cd618
use python None instead
milkshakeiii May 6, 2024
be8c8ff
add parens for clarity
milkshakeiii May 6, 2024
9bb2571
Merge branch 'main' into b332758928-stack-bug
milkshakeiii May 6, 2024
70c5d22
allow NA to match nan
milkshakeiii May 6, 2024
c06f492
fix comment
milkshakeiii May 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion bigframes/core/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -429,7 +429,10 @@ def _create_unpivot_labels_array(
for row_offset in range(len(former_column_labels)):
row_label = former_column_labels[row_offset]
row_label = (row_label,) if not isinstance(row_label, tuple) else row_label
row = {col_ids[i]: row_label[i] for i in range(len(col_ids))}
row = {
col_ids[i]: (row_label[i] if pandas.notnull(row_label[i]) else None)
for i in range(len(col_ids))
}
rows.append(row)

return ArrayValue.from_pyarrow(pa.Table.from_pylist(rows), session=self.session)
Expand Down
19 changes: 19 additions & 0 deletions tests/system/small/test_multiindex.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1191,3 +1191,22 @@ def test_explode_w_multi_index():
check_dtype=False,
check_index_type=False,
)


def test_column_multi_index_w_na_stack(scalars_df_index, scalars_pandas_df_index):
columns = ["int64_too", "int64_col", "rowindex_2"]
level1 = pandas.Index(["b", "c", "d"])
# Need resulting column to be pyarrow string rather than object dtype
level2 = pandas.Index([None, "b", "b"], dtype="string[pyarrow]")
multi_columns = pandas.MultiIndex.from_arrays([level1, level2])
bf_df = scalars_df_index[columns].copy()
bf_df.columns = multi_columns
pd_df = scalars_pandas_df_index[columns].copy()
pd_df.columns = multi_columns

pd_result = pd_df.stack()
bf_result = bf_df.stack().to_pandas()

# Pandas produces pd.NA, where bq dataframes produces NaN
pd_result["c"] = pd_result["c"].replace(pandas.NA, np.nan)
pandas.testing.assert_frame_equal(bf_result, pd_result, check_dtype=False)