Skip to content

chore: Change dev workflow to use uv (and pixi) both for lock and install #4166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeremyary merged 3 commits into from
May 8, 2024
Merged

chore: Change dev workflow to use uv (and pixi) both for lock and install #4166

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f60a51b
change dev workflow to use uv (and pixi)
tokoko May 1, 2024
ee07c7d
pin ibis versions
tokoko May 7, 2024
951c357
remove redundant make command
tokoko May 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ install-python-ci-dependencies-uv:
python setup.py build_python_protos --inplace

lock-python-ci-dependencies:
python -m piptools compile -U --extra ci --output-file sdk/python/requirements/py$(PYTHON)-ci-requirements.txt
uv pip compile --system --no-strip-extras setup.py --extra ci --output-file sdk/python/requirements/py$(PYTHON)-ci-requirements.txt

package-protos:
cp -r ${ROOT_DIR}/protos ${ROOT_DIR}/sdk/python/feast/protos
Expand All @@ -60,13 +60,15 @@ install-python:
python setup.py develop

lock-python-dependencies:
python -m piptools compile -U --output-file sdk/python/requirements/py$(PYTHON)-requirements.txt
uv pip compile --system --no-strip-extras setup.py --output-file sdk/python/requirements/py$(PYTHON)-requirements.txt

lock-python-dependencies-all:
pixi run --environment py39 --manifest-path infra/scripts/pixi/pixi.toml "python -m piptools compile -U --output-file sdk/python/requirements/py3.9-requirements.txt"
pixi run --environment py39 --manifest-path infra/scripts/pixi/pixi.toml "python -m piptools compile -U --extra ci --output-file sdk/python/requirements/py3.9-ci-requirements.txt"
pixi run --environment py310 --manifest-path infra/scripts/pixi/pixi.toml "python -m piptools compile -U --output-file sdk/python/requirements/py3.10-requirements.txt"
pixi run --environment py310 --manifest-path infra/scripts/pixi/pixi.toml "python -m piptools compile -U --extra ci --output-file sdk/python/requirements/py3.10-ci-requirements.txt"
pixi run --environment py39 --manifest-path infra/scripts/pixi/pixi.toml "uv pip compile --system --no-strip-extras setup.py --output-file sdk/python/requirements/py3.9-requirements.txt"
pixi run --environment py39 --manifest-path infra/scripts/pixi/pixi.toml "uv pip compile --system --no-strip-extras setup.py --extra ci --output-file sdk/python/requirements/py3.9-ci-requirements.txt"
pixi run --environment py310 --manifest-path infra/scripts/pixi/pixi.toml "uv pip compile --system --no-strip-extras setup.py --output-file sdk/python/requirements/py3.10-requirements.txt"
pixi run --environment py310 --manifest-path infra/scripts/pixi/pixi.toml "uv pip compile --system --no-strip-extras setup.py --extra ci --output-file sdk/python/requirements/py3.10-ci-requirements.txt"
pixi run --environment py311 --manifest-path infra/scripts/pixi/pixi.toml "uv pip compile --system --no-strip-extras setup.py --output-file sdk/python/requirements/py3.11-requirements.txt"
pixi run --environment py311 --manifest-path infra/scripts/pixi/pixi.toml "uv pip compile --system --no-strip-extras setup.py --extra ci --output-file sdk/python/requirements/py3.11-ci-requirements.txt"

benchmark-python:
FEAST_USAGE=False IS_TEST=True python -m pytest --integration --benchmark --benchmark-autosave --benchmark-save-data sdk/python/tests
Expand Down Expand Up @@ -359,9 +361,6 @@ kill-trino-locally:
install-protoc-dependencies:
pip install --ignore-installed protobuf==4.24.0 "grpcio-tools>=1.56.2,<2" mypy-protobuf==3.1.0

install-feast-ci-locally:
pip install -e ".[ci]"

# Docker

build-docker: build-feature-server-python-aws-docker build-feature-transformation-server-docker build-feature-server-java-docker
Expand Down
46 changes: 23 additions & 23 deletions docs/project/development-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,43 +123,43 @@ Note that this means if you are midway through working through a PR and rebase,
Setting up your development environment for Feast Python SDK / CLI:
1. Ensure that you have Docker installed in your environment. Docker is used to provision service dependencies during testing, and build images for feature servers and other components.
- Please note that we use [Docker with BuiltKit](https://docs.docker.com/develop/develop-images/build_enhancements/).
2. Ensure that you have `make`, Python (3.8 and above) with `pip`, installed.
2. Ensure that you have `make` and Python (3.9 or above) installed.
3. _Recommended:_ Create a virtual environment to isolate development dependencies to be installed
```sh
# create & activate a virtual environment
python -m venv venv/
source venv/bin/activate
```
4. Upgrade `pip` if outdated
```sh
pip install --upgrade pip
```
5. (M1 Mac only): Follow the [dev guide](https://github.com/feast-dev/feast/issues/2105)
6. Install pip-tools
```sh
pip install pip-tools
```
7. (Optional): Install Node & Yarn. Then run the following to build Feast UI artifacts for use in `feast ui`
4. (M1 Mac only): Follow the [dev guide](https://github.com/feast-dev/feast/issues/2105)
5. Install uv
It is recommended to use uv for managing python dependencies.
```sh
curl -LsSf https://astral.sh/uv/install.sh | sh
```
or
```ssh
pip install uv
```
6. (Optional): Install Node & Yarn. Then run the following to build Feast UI artifacts for use in `feast ui`
```
make build-ui
```
8. Install mysql (needed for ci dependencies)
7. (Optional) install pixi
pixi is necessary to run step 8 for all python versions at once.
```sh
brew install mysql
Copy link
Member

@franciscojavierarceo franciscojavierarceo May 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We removed this?

Copy link
Collaborator Author

@tokoko tokoko May 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was necessary for mysqlclient which was removed in #3925. pymysql doesn't require any native dependencies.

curl -fsSL https://pixi.sh/install.sh | bash
```
9. Install development dependencies for Feast Python SDK / CLI
8. (Optional): Recompile python lock files
If you make changes to requirements or simply want to update python lock files to reflect latest versioons.
```sh
pip install -e ".[dev]"
```

This will allow the installed feast version to automatically reflect changes to your local development version of Feast without needing to reinstall everytime you make code changes.

10. Compile the protubufs
Copy link
Member

@franciscojavierarceo franciscojavierarceo May 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we no longer need the compile the protos? if someone makes a Message change we will, no?

Copy link
Collaborator Author

@tokoko tokoko May 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do, it's just part of the make install-python-ci-dependencies-uv already

make lock-python-dependencies-all
Copy link
Member

@franciscojavierarceo franciscojavierarceo May 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

```
9. Install development dependencies for Feast Python SDK / CLI
This will install package versions from the lock file, install editable version of feast and compile protobufs.
```sh
make compile-protos-python
make install-python-ci-dependencies-uv
```

11. Spin up Docker Image
10. Spin up Docker Image
```sh
docker build -t docker-whale -f ./sdk/python/feast/infra/feature_servers/multicloud/Dockerfile .
```
Expand Down
Loading