Overwrite partitioned remote parquet files #21137

All three behaviours come from one function, CheckDirectory in physical_copy_to_file.cpp, plus a guard that depends on your version.

• Without PARTITION_BY, COPY writes a single object and never checks whether it exists; an S3 PUT simply replaces it. Nothing to enable.
• With PARTITION_BY, the target is a directory. CheckDirectory lists it and, if any file is present, requires either OVERWRITE (delete everything under the prefix, then write) or OVERWRITE_OR_IGNORE/APPEND, which skip the check entirely (v1.5.5, L416-L446).
• Up to v1.4.x, OVERWRITE on a remote path threw OVERWRITE is not supported for remote file systems because httpfs had no way to delete objects (v1.4.4, L392-L396). That guard was removed in #20487 once httpfs implemented RemoveFiles for S3 (duckdb-httpfs#181, s3fs.cpp on the v1.5 branch). Both shipped with v1.5.0 (2026-03-06).

So, from the shell:

-- v1.5.0 and later: removes the existing objects under the prefix, then writes
COPY tbl TO 's3://bucket/prefix' (FORMAT parquet, PARTITION_BY (year), OVERWRITE);

This needs delete permission on the prefix (s3:DeleteObject). On v1.4.x the only remote option is

COPY tbl TO 's3://bucket/prefix' (FORMAT parquet, PARTITION_BY (year), OVERWRITE_OR_IGNORE);

which overwrites objects whose names collide with the new run (year=2024/data_0.parquet, ...) and leaves everything else in place. That is fine when the partition layout and file count are the same on every run; otherwise stale files survive, which is why it is called "or ignore" rather than "overwrite".

This is from the source paths, not from a run against a bucket, so if OVERWRITE on 1.5.x fails for you the interesting part is the error text.