Conversation
|
the breaks in the test cases are due to the older arrow 26 coming from connector-x which is conflicting with the new arrow 37. raised sfu-db/connector-x#493 |
This serialization change looks good to me 👍
Yes, we can use the export from one of the crates, probably makes more sense to use the one exported from datafusion. |
| lto = true | ||
| codegen-units = 1 | ||
|
|
||
| [patch.crates-io] |
There was a problem hiding this comment.
@chitralverma we can patch connectorx to use our own temp fork at https://github.com/roapi/connector-x if it's taking a long time for upstream to merge your PR.
There was a problem hiding this comment.
Sounds good, can you please sync that fork with the latest upstream
There was a problem hiding this comment.
Just sent you an invite for write access to the repo ;)
|
FYI, I just cut deltalake 0.10.0 which has the datafusion 23 and arrow 37 dependencies referenced, roapi might be able to move away from git sha1s now? :) |
Awesome, I was waiting for this 😁 |
@houqp I tried this but apparently, datafusion doesn't export object_store for external use. So we will have to add it explicitly |
| [dependencies] | ||
| # pulling arrow-schema manually to enable the serde feature. | ||
| # TODO: add serde feature in datafusion to avoid this workaround | ||
| arrow-schema = { version ="26", features = ["serde"] } |
There was a problem hiding this comment.
@chitralverma we might need to set the arrow-schema version to 38 so it remains the same as the version pinned in datafusion to fix the build issue, see apache/datafusion@fc5d67a#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L51
|
Thanks @chitralverma again for the patch, I pushed a fix based off your branch with connectorx upgrade in #279 |
Thanks @houqp |
3 participants
Which issue does this PR close?
Closes #263
Related PR #264
What changes are included in this PR?
This PR upgrades various dependencies (see list below) of roapi to the latest supported versions.
14->23)26->37)0.5.4->0.5.6)internal patch->0.11.0latest tag)0.27->0.33)patched 0.5.0->patched 0.10.0)Other than this, this PR includes
ObjectStoreProvider,&str->Stringto fix lifetime issues.Are there any user-facing changes?
Mostly these are internal upgrades, so user-facing changes should be none.
Point of consideration for reviewers
Cargo.tomlexplicitly includes theobject_storedependency which is currently coming from bothdelta-rsanddatafusion. So do you think removing this and using it transitively instead makes sense?