Skip to content

feat: add DataFrame.writeCsv with CsvWriteOptions #38

Description

@andygrove

Is your feature request related to a problem or challenge?

DataFrame.writeParquet shipped in #27, but the symmetric CSV writer is
missing. DataFusion exposes DataFrame::write_csv, which is commonly
used for ad-hoc exports.

Describe the solution you'd like

  • Add a CsvWriteOptions value class covering the knobs that
    DataFrameWriteOptions + CSV-specific options expose
    (single-file output, partition columns, header, delimiter, quote,
    escape, null-value string, compression).
  • Add proto/csv_write_options.proto, following the convention
    established by parquet_write_options.proto in feat(write): add DataFrame.writeParquet with ParquetWriteOptions #27.
  • Expose DataFrame.writeCsv(path[, options]).
  • Cover with tests in the spirit of DataFrameWriteParquetTest
    (single-file, compression, retain-after-write semantics).

Describe alternatives you've considered

COPY (SELECT * FROM df) TO '…' via SQL works but requires a
registered name and a SQL detour.

Additional context

Aligns the writer story: with this and the JSON writer, we cover the
three formats DataFusion's DataFrame API can write today
(write_parquet, write_csv, write_json).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions