WIP Push Docker image and build from it in sub-stages #2013
Conversation
Revamping Travis and Docker setup introducing a `Makefile`. The idea is to move the CI complexity from .travis.yml to `Makefile`. That makes a single entry point via `make` command and reproducible builds via Docker. It makes it easy to run some commands outside docker, such as: ```sh make testapps/python3/armeabi-v7a ``` Or the same command inside docker: ```sh make docker/run/make/testapps/python3/armeabi-v7a ``` This pull request also starts introducing some docker layer cache optimization as needed by kivy#2009 to speed up docker pull/push and rebuilds from cache. It also introduces other Docker images good practices like ordering dependencies alphabetically or always enforcing `apt update` prior install, refs: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/ Subsequent pull requests would simplify the process furthermore and leverage the cache to speed up builds.
The `docker build` is now running alongside the `tox` testing as part of the `pre checks` stage. It also fully leverages the docker image cache. The newly built image is pushed to be available for subsequent build stages and pull requests. Two environment variables `DOCKER_USERNAME` and `DOCKER_PASSWORD` got added to the Travis UI for making image pushing possible. Both `push` and `pull` features fail silently with `|| true` as they should be optional for a valid build. `TRAVIS_PULL_REQUEST` and `TRAVIS_BRANCH` are built-in Travis variables. Note that we're passing variables to `Makefile` explicitly on `make` call rather than using the `--environment-overrides` flag. While it's more verbose, it makes it easier to follow what's being used and when. With this change, `Python 3 arm64-v8a` the longest `testapps` build that was taking ~26 minutes to complete is now taking ~20 minutes when hitting docker layers cache (most of the time). The shorter build `Rebuild updated recipes` was taking ~9 minutes and is now down to ~3 minutes. Closes kivy#2009
|
Build flow and time before the PR: And after (on my fork): |
AndreMiras
changed the title
|
Just changed to |
|
OK so let's not deal with |
1 participant
This is a follow-up for #2011 it also supersedes it as it's built on top. See description below:
The
docker buildis now running alongside thetoxtesting as part ofthe
pre checksstage. It also fully leverages the docker image cache.The newly built image is pushed to be available for subsequent build
stages and pull requests.
Two environment variables
DOCKER_USERNAMEandDOCKER_PASSWORDgotadded to the Travis UI for making image pushing possible. Both
pushand
pullfeatures fail silently with|| trueas they should beoptional for a valid build.
TRAVIS_PULL_REQUESTandTRAVIS_BRANCHare built-in Travis variables.Note that we're passing variables to
Makefileexplicitly onmakecall rather than using the
--environment-overridesflag. While it'smore verbose, it makes it easier to follow what's being used and when.
With this change,
Python 3 arm64-v8athe longesttestappsbuild thatwas taking ~26 minutes to complete is now taking ~20 minutes when
hitting docker layers cache (most of the time). The shorter build
Rebuild updated recipeswas taking ~9 minutes and is now down to ~3minutes.
Closes #2009