DuckDB httpfs extension for .db files

Question

I'm experimenting with DuckDB (0.10.1) and the httpfs extension. Documentation states

With the httpfs extension, it is possible to directly query files over the HTTP(S) protocol. This works for all files supported by DuckDB or its various extensions, and provides read-only access.

However the only examples I can find are for Parquet or CSV files. When I try to query a DuckDB .db file over https the errors indicate I'm doing something wrong:

from 'https://.../file.db';
# Catalog Error: Table with name https://.../file.db does not exist!
# Did you mean "sqlite_master"?
# LINE 1: from 'https://...

and

from 's3://.../file.db';
# Catalog Error: Table with name s3://.../file.db does not exist!
# Did you mean "pg_attrdef"?
# LINE 1: from 's3://...

I have verified using curl and s3 that both URLs are publicly accessible. I can successfully query a Parquet file in the same S3 bucket using the httpfs extension, so I'm confident the extension is working.

Given that the errors refer to missing tables, is there a specific syntax to identify the name of a table within that remote data source? Or is the documentation wrong, and not all file types are supported?

Sam Ansmink · Accepted Answer · 2024-04-02 13:16:17Z

3

You can use attach to achieve this.

For example:

ATTACH 'https://herp.derp/my.db' AS db (READONLY 1);
FROM db.some_table;

Note that this part of DuckDB is being actively worked on, with s3 support coming up and more performance optimizations being planned here. Also note that currently attaching DB files over HTTP is read-only.

answered Apr 2, 2024 at 13:16

Sam Ansmink

1212 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

tomfumb Over a year ago

Thanks - this did work, but was way slower than I expected :( Queries that would be quick on a remote parquet file were slow on a remote .db file, or simply bombed with a segmentation fault

Sam Ansmink Over a year ago

Yea this is not as well optimized as the remote parquet reading at the moment.

Collectives™ on Stack Overflow

DuckDB httpfs extension for .db files

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related