Skip to content

Commit 3d211b4

Browse files
vivian-rookvivian-rook
authored
Merge branch 'master' into lsof
2 parents 19e8133 + 0b164fc commit 3d211b4

37 files changed

+546
-751
lines changed

‎.github/workflows/jobber.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: build and push jobber
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- images/jobber/**
7+
8+
jobs:
9+
build-and-push:
10+
name: build and push jobber
11+
uses: toolforge/github-actions/.github/workflows/build-and-push.yaml@build-and-push-v1
12+
with:
13+
imagename: jobber
14+
secrets:
15+
quay_user: ${{ secrets.QUAY_USER }}
16+
quay_password: ${{ secrets.QUAY_PASSWORD }}

‎.github/workflows/lint.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
name: Python lint
2+
3+
on:
4+
pull_request:
5+
6+
jobs:
7+
flake8_py3:
8+
runs-on: ubuntu-latest
9+
steps:
10+
- uses: actions/checkout@master
11+
- name: Setup Python
12+
uses: actions/setup-python@v1
13+
with:
14+
python-version: 3.8
15+
- name: Install tox
16+
run: pip install tox
17+
- name: Run tox
18+
run: |
19+
tox
20+
- name: Install kubeval
21+
run: |
22+
mkdir kubeval
23+
cd kubeval
24+
curl -Lf https://github.com/instrumenta/kubeval/releases/download/0.15.0/kubeval-linux-amd64.tar.gz | tar xzf -
25+
chmod +x kubeval
26+
- name: Run kubeval
27+
run: |
28+
export KUBERNETES_VERSION=1.16.0
29+
./kubeval/kubeval -v $KUBERNETES_VERSION --strict manifests/*
30+
- uses: azure/setup-helm@v1
31+
- name: Helm lint
32+
run: |
33+
helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
34+
helm repo add bitnami https://charts.bitnami.com/bitnami
35+
helm dependency update paws/
36+
helm lint paws/

‎.github/workflows/nbserve.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: build and push nbserve
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- images/nbserve/**
7+
8+
jobs:
9+
build-and-push:
10+
name: build and push nbserve
11+
uses: toolforge/github-actions/.github/workflows/build-and-push.yaml@build-and-push-v1
12+
with:
13+
imagename: nbserve
14+
secrets:
15+
quay_user: ${{ secrets.QUAY_USER }}
16+
quay_password: ${{ secrets.QUAY_PASSWORD }}

‎.github/workflows/paws-hub.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: build and push paws-hub
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- images/paws-hub/**
7+
8+
jobs:
9+
build-and-push:
10+
name: build and push paws-hub
11+
uses: toolforge/github-actions/.github/workflows/build-and-push.yaml@build-and-push-v1
12+
with:
13+
imagename: paws-hub
14+
secrets:
15+
quay_user: ${{ secrets.QUAY_USER }}
16+
quay_password: ${{ secrets.QUAY_PASSWORD }}

‎.github/workflows/renderer.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: build and push renderer
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- images/renderer/**
7+
8+
jobs:
9+
build-and-push:
10+
name: build and push renderer
11+
uses: toolforge/github-actions/.github/workflows/build-and-push.yaml@build-and-push-v1
12+
with:
13+
imagename: renderer
14+
secrets:
15+
quay_user: ${{ secrets.QUAY_USER }}
16+
quay_password: ${{ secrets.QUAY_PASSWORD }}

‎.github/workflows/singleuser.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
name: build and push singleuser
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- images/singleuser/**
7+
8+
9+
jobs:
10+
build-and-push:
11+
name: build and push singleuser
12+
uses: toolforge/github-actions/.github/workflows/build-and-push.yaml@build-and-push-v1
13+
with:
14+
imagename: singleuser
15+
secrets:
16+
quay_user: ${{ secrets.QUAY_USER }}
17+
quay_password: ${{ secrets.QUAY_PASSWORD }}

‎.github/workflows/update-container-tags.yaml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
name: update container tags
2+
3+
on:
4+
pull_request:
5+
6+
jobs:
7+
update-container-tags:
8+
runs-on: ubuntu-latest
9+
steps:
10+
- uses: actions/checkout@v2
11+
with:
12+
token: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
13+
14+
- name: git fetch
15+
run: |
16+
git fetch
17+
18+
# Github actions doesn't have much in the way of loops. Using a matrix results
19+
# in multiple separate jobs being run, which will try to overwrite one another.
20+
# Any kind of loop shaped thing would be more elegant than this. Until then,
21+
# when a container is added copy one of these and change then name in all four
22+
# places
23+
- name: update values.yaml for renderer
24+
run: |
25+
if [[ $(git diff remotes/origin/master -- images/renderer/) ]]; then
26+
sed -i 's/tag: .* # renderer tag managed by github actions$/tag: pr-${{ github.event.number }} # renderer tag managed by github actions/' paws/values.yaml
27+
fi
28+
29+
- name: update values.yaml for nbserve
30+
run: |
31+
if [[ $(git diff remotes/origin/master -- images/nbserve/) ]]; then
32+
sed -i 's/tag: .* # nbserve tag managed by github actions$/tag: pr-${{ github.event.number }} # nbserve tag managed by github actions/' paws/values.yaml
33+
fi
34+
35+
- name: update values.yaml for paws-hub
36+
run: |
37+
if [[ $(git diff remotes/origin/master -- images/paws-hub/) ]]; then
38+
sed -i 's/tag: .* # paws-hub tag managed by github actions$/tag: pr-${{ github.event.number }} # paws-hub tag managed by github actions/' paws/values.yaml
39+
fi
40+
41+
- name: update values.yaml for jobber
42+
run: |
43+
if [[ $(git diff remotes/origin/master -- images/jobber/) ]]; then
44+
sed -i 's/tag: .* # jobber tag managed by github actions$/tag: pr-${{ github.event.number }} # jobber tag managed by github actions/' paws/values.yaml
45+
fi
46+
47+
- name: update values.yaml for singleuser
48+
run: |
49+
if [[ $(git diff remotes/origin/master -- images/singleuser/) ]]; then
50+
sed -i 's/tag: .* # singleuser tag managed by github actions$/tag: pr-${{ github.event.number }} # singleuser tag managed by github actions/' paws/values.yaml
51+
fi
52+
53+
- uses: EndBug/add-and-commit@v7
54+
with:
55+
add: 'paws/values.yaml'
56+
author_name: Github Action
57+
author_email: auto@github.com
58+
branch: ${{ github.head_ref }}
59+
message: 'auto update of ${{ inputs.imagename }} tag'
60+
pull: --rebase --autostash

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,6 @@ __pycache__/
55
.mypy_cache/
66
charts/
77
requirements.lock
8+
Chart.lock
89
dev-values.yaml
10+
*.swp

‎.travis.yml‎

Lines changed: 0 additions & 63 deletions
This file was deleted.

‎README.md‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,17 +16,19 @@ to trigger a pull request or using a fork to set up a pull request.
1616
### Settings up a development environment
1717

1818
It is possible to run a fully-functioning PAWS system inside [minikube](https://minikube.sigs.k8s.io/docs/)! You don't need
19-
access to the secrets.yaml file to do it either, since the defaults mostly support it.
19+
access to the secrets.yaml file to do it either, since the defaults mostly support it. At this time, you need to
20+
set it up with a cluster version before 1.22, most likely.
2021

21-
You will need to install minikube and [helm](https://helm.sh) on your system. When you are confident those are working,
22+
You will need to install minikube (tested on minikube 1.23) and [helm](https://helm.sh) and kubectl on your system. When you are confident those are working, start minikube with:
23+
- `minikube start --kubernetes-version=v1.20.11`
24+
- `minikube addons enable ingress`
25+
(from the top level of this repo):
2226
install the dependencies for the PAWS dev environment with these steps:
2327
- `helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/`
2428
- `helm repo add bitnami https://charts.bitnami.com/bitnami`
2529
- `helm dep up paws/`
26-
27-
Then, you need to create a new namespace for paws (`kubectl create namespace paws-dev` for instance) and then install into it
28-
with helm (from the top level of this repo):
29-
`helm -n paws-dev install dev paws/`
30+
- `kubectl create namespace paws-dev`
31+
- `helm -n paws-dev install dev paws/ --timeout=50m`
3032

3133
The rest of the setup instructions will display on screen as long as the install is successful.
3234
Please refer to the helm documentation from there.
@@ -40,6 +42,16 @@ If minikube is acting weird, it might be worth it to upgrade minikube or even to
4042
increase the default memory:
4143
`minikube config set memory 4096`
4244

45+
#### Working with images
46+
There are 8 images that are part of PAWS, in particular in the images/ directory. If you start a dev environment, it will pull those images from quay.io by default, just like in Wikimedia Cloud Services. If you are making changes to the images and testing those locally, you'll need to build them and tag them for your local environment, possibly setting them in your local values file with the tags you set.
47+
48+
If you are using minikube, you need to make sure you are using minikube's docker, not your system's docker with `eval $(minikube docker-env)`. Now your docker commands will operate on the minikube environment.
49+
50+
For example, let's say you wanted to update the singleuser image (which is the actual notebook server image):
51+
- `cd images/singleuser`
52+
- `docker build -t tag-you-are-going-to-use:whatever .`
53+
54+
And then you should have the image with a tag of `tag-you-are-going-to-use:whatever` that you could edit into your values.yaml file for local helm work. If you were aiming to aggressively push a tag for deployment, dodging around the CI system to do so, you'd tag with `quay.io/wikimedia-paws-prod/singleuser:latest`, which will cause a later `docker push quay.io/wikimedia-paws-prod/singleuser:latest` to actually push that tag directly to the repo for deployment in Cloud Services. Don't do that unless you really know what you are doing.
4355
## Useful libraries
4456
### Accessing Database Replicas With Pandas and Sqlalchemy
4557

0 commit comments

Comments
 (0)