chore: Install python dependencies with uv in workflows #4086
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: tokoko <togurg14@freeuni.edu.ge>
Signed-off-by: tokoko <togurg14@freeuni.edu.ge>
Signed-off-by: tokoko <togurg14@freeuni.edu.ge>
Signed-off-by: tokoko <togurg14@freeuni.edu.ge>
tokoko
changed the title
Signed-off-by: tokoko <togurg14@freeuni.edu.ge>
Contributor
Maybe this helps somehow with the caches: "The cache action first searches for cache hits for key and the cache version in the branch containing the workflow run. If there is no hit, it searches for restore-keys and the version. If there are still no hits in the current branch, the cache action retries same steps on the default branch" I think it solves the problem with the caches because in a pull request the merging is usually into master in which the artifacts should already be in, don't you think ? |
tmihalac
reviewed
Apr 10, 2024
Collaborator
Author
|
@tmihalac that might be a good idea, unit tests are never run on master, but integration tests are and unit test jobs might be able to reuse caches written by integration tests. Not sure what will happen when two jobs try to update the cache with the same key at the same time though. |
Signed-off-by: tokoko <togurg14@freeuni.edu.ge>
Signed-off-by: tokoko <togurg14@freeuni.edu.ge>
tmihalac
reviewed
Apr 10, 2024
Signed-off-by: tokoko <togurg14@freeuni.edu.ge>
Collaborator
Author
|
@tmihalac One concern I still have is that looks like some people's experience with caching and uv on github is that often it might actually slow things down (see here). I have reenabled caches for all workflows except the linter one. We can use that to compare the install runtimes and then make a decision whether we should leave caching in or not... does that sound okay? |
Contributor
|
Sounds good to me. To me it seems the tests take shorter time, do you think the same ? |
Signed-off-by: tokoko <togurg14@freeuni.edu.ge>
Collaborator
Author
|
I'm not too sure actually, just from a sample size of 1, P.S. For some reason everything's slower on mac vms 😄 |
Collaborator
Author
|
@jeremyary I think this is safe to merge |
Member
|
Seems like a big change... CC @HaoXuAI |
Collaborator
Author
|
fwiw, even though |
Collaborator
|
Is the cache totally not making any benefits for build? |
Collaborator
Author
|
@HaoXuAI sorry, PR description is misleading now, I have added cache back in to monitor that. tldr version is that, right now it's probably a net loss actually, we are using an older version of cache action, it's slow to upload and sometimes just fails to find the cache even if it exists. With upgraded cache action and pip, it would definitely be of value, no question about it. The question is whether that still is the case with uv instead of pip. uv is much much faster than pip and to me it looks like cache often brings no real benefit, in other words time spent on cache upload/download is roughly the same as a fresh uv install. The current version of this PR disables caching for a linter job only and we can use that to make a final decision later. |
Collaborator
Author
|
This run is a good example. cache step seems to work at first, but decides the cache is invalid for some reason. pip install step takes 2.5 minutes without cache and then the job proceeds to waste almost 6 minutes to upload the directory again. For comparison uv install (without cache) is basically always under a minute. |
HaoXuAI
reviewed
Apr 23, 2024
| pip install --no-deps -e . | ||
| python setup.py build_python_protos --inplace | ||
|
|
||
| install-python-ci-dependencies-uv: |
Collaborator
There was a problem hiding this comment.
i'm not familiar with uv. but using it to install in the CI, will it have a different result with direct pip install?
Collaborator
Author
There was a problem hiding this comment.
For all intents and purposes, no, it's intended to be a drop-in replacement. There's of course some minor differences, pip tends to be more lenient towards packages with invalid metadata for example, version resolution (which we aren't changing in this pr, but I'll do in a follow-up) might also be negligibly different because they are using different resolvers with different heuristics. But at the end of the day, it shouldn't matter for us.
HaoXuAI
approved these changes
Apr 23, 2024
Collaborator
HaoXuAI
left a comment
HaoXuAI
left a comment
There was a problem hiding this comment.
LGTM.
like the idea of upgrading the python env tool.
lokeshrangineni
pushed a commit
to lokeshrangineni/feast
that referenced
this pull request
Apr 25, 2024
* install dependencies in unit-tests with uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * install dependencies in unit-tests with uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * enable caching, change linter job Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change local integration tests to uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change all installs to uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try adding uv cache Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * fix lambda cache step name Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * reenable caches for uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove dangling cache step Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com>
jeremyary
added a commit
that referenced
this pull request
Apr 25, 2024
* chore: Move feast install to docker build in java it tests (#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove commented out lines in compose file Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * make local compose mode default Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * limit COPY contents Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * include pyproject.toml in dockerfile Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change links to depends_on Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try updating setup-python to v5 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * pin macos image to macos-12 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * force rerun Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * chore: Incorporate release 0.37.1 to master (#4127) * chore(release): release 0.37.1 ## [0.37.1](v0.37.0...v0.37.1) (2024-04-17) ### Bug Fixes * Pgvector patch ([#4108](#4108)) ([1a1f0b1](1a1f0b1)) ### Reverts * Reverts "fix: Using version args to install the correct feast version" ([#4112](#4112)) ([d5ded69](d5ded69)), closes [#3953](#3953) * chore: Move feast install to docker build in java it tests (#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove commented out lines in compose file Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * make local compose mode default Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * limit COPY contents Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * include pyproject.toml in dockerfile Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change links to depends_on Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try updating setup-python to v5 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * pin macos image to macos-12 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * force rerun Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co> Co-authored-by: Tornike Gurgenidze <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * chore: Install python dependencies with uv in workflows (#4086) * install dependencies in unit-tests with uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * install dependencies in unit-tests with uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * enable caching, change linter job Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change local integration tests to uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change all installs to uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try adding uv cache Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * fix lambda cache step name Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * reenable caches for uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove dangling cache step Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * feat: Make arrow primary interchange for online ODFV execution (#4143) * rewrite online flow to use transform_arrow Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * fix transformation server Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * feat: Feast/IKV online store documentation (#4146) * feat: Feast/IKV online store documentation Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> * functionality matric Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> * more changes Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> * mount dir Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> --------- Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * fix: Default value is not set in Redis connection string using environment variable (#4136) Removed documentation of Redis connection string supporting default values when using environment variables as it isn't supported Fixes #3669 Signed-off-by: Theodor Mihalache <tmihalac@redhat.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * fix: Make sure schema is used when calling `get_table_query_string` method for Snowflake datasource (#4131) * Fix get_table_query_string method for Snowflake datasource Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> * Add quotes to table string Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> --------- Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * fix: Change checkout action back to v3 from v5 which isn't released yet (#4147) Signed-off-by: Jeremy Ary <jary@redhat.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * changed the code the way mysql container is initialized. Trying to fix the issue - #4128 Also going to check if this change will be resolved in the github actions as well. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> Signed-off-by: Theodor Mihalache <tmihalac@redhat.com> Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> Signed-off-by: Jeremy Ary <jary@redhat.com> Co-authored-by: Tornike Gurgenidze <togurg14@freeuni.edu.ge> Co-authored-by: Jeremy Ary <jary@redhat.com> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co> Co-authored-by: Pushkar Gupta <pushkar.moi@gmail.com> Co-authored-by: Theodor Mihalache <84387487+tmihalac@users.noreply.github.com> Co-authored-by: Tom Steenbergen <41334387+TomSteenbergen@users.noreply.github.com>
lokeshrangineni
added a commit
to lokeshrangineni/feast
that referenced
this pull request
Apr 25, 2024
…dev#4140) * chore: Move feast install to docker build in java it tests (feast-dev#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove commented out lines in compose file Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * make local compose mode default Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * limit COPY contents Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * include pyproject.toml in dockerfile Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change links to depends_on Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try updating setup-python to v5 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * pin macos image to macos-12 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * force rerun Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * chore: Incorporate release 0.37.1 to master (feast-dev#4127) * chore(release): release 0.37.1 ## [0.37.1](feast-dev/feast@v0.37.0...v0.37.1) (2024-04-17) ### Bug Fixes * Pgvector patch ([feast-dev#4108](feast-dev#4108)) ([1a1f0b1](feast-dev@1a1f0b1)) ### Reverts * Reverts "fix: Using version args to install the correct feast version" ([feast-dev#4112](feast-dev#4112)) ([d5ded69](feast-dev@d5ded69)), closes [feast-dev#3953](feast-dev#3953) * chore: Move feast install to docker build in java it tests (feast-dev#4126) * chore: Move feast install to docker build in java it tests Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove commented out lines in compose file Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * make local compose mode default Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * limit COPY contents Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove requirements.txt from java tests docker image Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * include pyproject.toml in dockerfile Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change links to depends_on Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try updating setup-python to v5 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * pin macos image to macos-12 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * force rerun Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co> Co-authored-by: Tornike Gurgenidze <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * chore: Install python dependencies with uv in workflows (feast-dev#4086) * install dependencies in unit-tests with uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * install dependencies in unit-tests with uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * enable caching, change linter job Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change local integration tests to uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * change all installs to uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * try adding uv cache Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * fix lambda cache step name Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * reenable caches for uv Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * remove dangling cache step Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * feat: Make arrow primary interchange for online ODFV execution (feast-dev#4143) * rewrite online flow to use transform_arrow Signed-off-by: tokoko <togurg14@freeuni.edu.ge> * fix transformation server Signed-off-by: tokoko <togurg14@freeuni.edu.ge> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * feat: Feast/IKV online store documentation (feast-dev#4146) * feat: Feast/IKV online store documentation Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> * functionality matric Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> * more changes Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> * mount dir Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> --------- Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * fix: Default value is not set in Redis connection string using environment variable (feast-dev#4136) Removed documentation of Redis connection string supporting default values when using environment variables as it isn't supported Fixes feast-dev#3669 Signed-off-by: Theodor Mihalache <tmihalac@redhat.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * fix: Make sure schema is used when calling `get_table_query_string` method for Snowflake datasource (feast-dev#4131) * Fix get_table_query_string method for Snowflake datasource Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> * Add quotes to table string Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> --------- Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * fix: Change checkout action back to v3 from v5 which isn't released yet (feast-dev#4147) Signed-off-by: Jeremy Ary <jary@redhat.com> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * changed the code the way mysql container is initialized. Trying to fix the issue - feast-dev#4128 Also going to check if this change will be resolved in the github actions as well. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * reformatted the file to resolve lint error. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> --------- Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Signed-off-by: Pushkar Gupta <pushkar.moi@gmail.com> Signed-off-by: Theodor Mihalache <tmihalac@redhat.com> Signed-off-by: TomSteenbergen <tomsteenbergen1995@gmail.com> Signed-off-by: Jeremy Ary <jary@redhat.com> Co-authored-by: Tornike Gurgenidze <togurg14@freeuni.edu.ge> Co-authored-by: Jeremy Ary <jary@redhat.com> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co> Co-authored-by: Pushkar Gupta <pushkar.moi@gmail.com> Co-authored-by: Theodor Mihalache <84387487+tmihalac@users.noreply.github.com> Co-authored-by: Tom Steenbergen <41334387+TomSteenbergen@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it: