feat: support postgresql - #286

@houqp Is this going to merge? I aslo have interesting about this

@houqp We are very interested in this feature! When can we expect its implementation?

sorry for the delay, will take a look at this later today

@holicc do you need help fixing the CI? I was planning to dig into that, but don't want to step on your toes.

@holicc do you need help fixing the CI? I was planning to dig into that, but don't want to step on your toes.

@houqp Yes, I'm not sure how to fix it.

@holicc this is because the gcs setup in CI was failing, for some reasons, the curl command was not able to store the parquet file there. Looking into it more now.

hopefully #291 should fix it.

@holicc could you rebase to get a clean CI run? I have fixed the main branch.

OK, now we are back to MacOS openssl cross build issues :(

@holicc I think the quickest fix is to disable the postgres feature in the mac build job:

OK, now we are back to MacOS openssl cross build issues :(

@holicc I think the quickest fix is to disable the postgres feature in the mac build job:

roapi/.github/workflows/build.yml

Line 292 in b8b772a

cargo build --bin roapi --features database --target aarch64-apple-darwin

.

It appears that we need to install openssl in the build environment.

• OpenSSL library not found rust-openssl/rust-openssl#855
• https://ma.ttias.be/could-not-find-directory-of-openssl-installation/

@houqp I don't know how to trigger the flow. Can you help me out?

@holicc I was suggesting we don't use the --features database in the macos crossbuild job, but instead build explicitly with the following flags to exclude postgres for mac for now:

• database-sqlite
• database-mysql

Cross building openssl on mac is going to be a major undertake unfortunately. So we can look into fixing that later.

@holicc I was suggesting we don't use the --features database in the macos crossbuild job, but instead build explicitly with the following flags to exclude postgres for mac for now:

• database-sqlite
• database-mysql

Cross building openssl on mac is going to be a major undertake unfortunately. So we can look into fixing that later.

My bad. I thought it was a simple problem.

Thank you @holicc !

Recently, I found that we can simplify the code base by using the get_arrow API from connectorx.
@houqp Do you think we should modify the code like this?

impl DatabaseLoader {
        pub fn to_mem_table(
            &self,
            t: &TableSource,
        ) -> Result<datafusion::datasource::MemTable, ColumnQError> {
            debug!("loading database table data...");
            let queries = CXQuery::naked(format!("SELECT * FROM {}", t.name));
            let source = SourceConn::try_from(t.get_uri_str())
                .map_err(|e| ColumnQError::Database(e.to_string()))?;
            let destination = connectorx::get_arrow::get_arrow(&source, None, &[queries])
                .map_err(|e| ColumnQError::Database(e.to_string()))?;
            Ok(datafusion::datasource::MemTable::try_new(
                destination.arrow_schema(),
                vec![destination.arrow().unwrap()],
            )?)
        }
    }

Yes, let's do that @holicc !