Skip to content

ci: Add a nightly CI job for integration tests #2652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 4 commits into from
Jul 12, 2022
Merged

ci: Add a nightly CI job for integration tests #2652

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9aacb2e
Remove 3.7 wheels
kevjumba Jul 11, 2022
b555db8
ci: Add a nightly CI job for integration tests
achals May 9, 2022
04a362e
Update and rebase
kevjumba Jul 11, 2022
9b5027a
Fix
kevjumba Jul 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/build_wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ jobs:
uses: pypa/cibuildwheel@v2.7.0
env:
CIBW_BUILD: "cp3*_x86_64"
CIBW_SKIP: "cp36-* *-musllinux_x86_64 cp310-macosx_x86_64"
CIBW_SKIP: "cp36-* cp37-* *-musllinux_x86_64 cp310-macosx_x86_64"
CIBW_ARCHS: "native"
CIBW_ENVIRONMENT: >
COMPILE_GO=True PATH=$PATH:/usr/local/go/bin
Expand Down Expand Up @@ -150,7 +150,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, macos-10.15 ]
python-version: [ "3.7", "3.8", "3.9", "3.10"]
python-version: [ "3.8", "3.9", "3.10"]
from-source: [ True, False ]
env:
# this script is for testing servers
Expand Down
140 changes: 140 additions & 0 deletions .github/workflows/nightly-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
name: nightly-ci

on:
schedule:
- cron: '00 08 * * *' # early morning 08:00 AM UTC, which is 1 AM PST/4 AM EST.

# concurrency is currently broken, see details https://github.com/actions/runner/issues/1532
#concurrency:
# group: pr-integration-tests-${{ github.event.pull_request.number }}
# cancel-in-progress: true

jobs:
check_date:
runs-on: ubuntu-latest
name: Check latest commit
outputs:
WAS_EDITED: ${{ steps.check_date.outputs.WAS_EDITED }}
steps:
- uses: actions/checkout@v2
with:
ref: develop
- id: check_date
name: Check if there were commits in the last day
if: ${{ github.event_name == 'schedule' }}
run: echo '::set-output name=WAS_EDITED::'$(test -n "$(git log --format=%H --since='24 hours ago')" && echo 'true' || echo 'false')

integration-test-python:
needs: [check_date]
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: [ "3.8" ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
PYTHON: ${{ matrix.python-version }}
services:
redis:
image: redis
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v2
with:
# pull_request_target runs the workflow in the context of the base repo
# as such actions/checkout needs to be explicit configured to retrieve
# code from the PR.
ref: refs/pull/${{ github.event.pull_request.number }}/merge
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v2
id: setup-python
with:
python-version: ${{ os.PYTHON }}
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.18.0
- name: Set up gcloud SDK
uses: google-github-actions/setup-gcloud@v0
with:
project_id: ${{ secrets.GCP_PROJECT_ID }}
service_account_key: ${{ secrets.GCP_SA_KEY }}
export_default_credentials: true
- name: Use gcloud CLI
run: gcloud info
- name: Set up AWS SDK
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Use AWS CLI
run: aws sts get-caller-identity
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<22.1"
- name: Get pip cache dir
id: pip-cache
run: |
echo "::set-output name=dir::$(pip cache dir)"
- name: pip cache
uses: actions/cache@v2
with:
path: |
${{ steps.pip-cache.outputs.dir }}
/opt/hostedtoolcache/Python
/Users/runner/hostedtoolcache/Python
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev
- name: Install apache-arrow on macos
if: matrix.os == 'macOS-latest'
run: brew install apache-arrow
- name: Install dependencies
run: make install-python-ci-dependencies
- name: Setup Redis Cluster
run: |
docker pull vishnunair/docker-redis-cluster:latest
docker run -d -p 6001:6379 -p 6002:6380 -p 6003:6381 -p 6004:6382 -p 6005:6383 -p 6006:6384 --name redis-cluster vishnunair/docker-redis-cluster
- name: Test python
if: ${{ always() }} # this will guarantee that step won't be canceled and resources won't leak
env:
FEAST_SERVER_DOCKER_IMAGE_TAG: ${{ needs.build-docker-image.outputs.DOCKER_IMAGE_TAG }}
FEAST_USAGE: "False"
IS_TEST: "True"
SNOWFLAKE_CI_DEPLOYMENT: ${{ secrets.SNOWFLAKE_CI_DEPLOYMENT }}
SNOWFLAKE_CI_USER: ${{ secrets.SNOWFLAKE_CI_USER }}
SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }}
SNOWFLAKE_CI_ROLE: ${{ secrets.SNOWFLAKE_CI_ROLE }}
SNOWFLAKE_CI_WAREHOUSE: ${{ secrets.SNOWFLAKE_CI_WAREHOUSE }}
run: pytest -n 8 --cov=./ --cov-report=xml --color=yes sdk/python/tests --integration --durations=5 --timeout=1200 --timeout_method=thread
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.xml
flags: integrationtests
env_vars: OS,PYTHON
fail_ci_if_error: true
verbose: true
2 changes: 1 addition & 1 deletion .github/workflows/pr_integration_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,4 +191,4 @@ jobs:
flags: integrationtests
env_vars: OS,PYTHON
fail_ci_if_error: true
verbose: true
verbose: true