Skip to content

Commit b5fa284

Browse files
rtilorenjohnson
andauthored
chore: Add wikibase-lts and deploy-lts products (#815)
* test: get current mediawiki version from api * chore: Add wikibase-lts and deploy-lts products * chore: Prepare initial LTS product releases * chore: Update nx to see the deploy-lts package/product * chore: Add LTS product variants as options to GHA Create a WBS Release * chore: Removes -lts post fix from the image name published to Dockerhub * wip: wikibase/deploy lts build/test/publish * fix: build wikibase-lts in ci * style: lint * fix: test deploy-lts * fix: deploy-lts test env config * fix(test): re-enable disabled repo and extensions specs * refactor(test/cli.ts): simplify suite names export * chore(test): remove unused import from deploy-lts.conf.ts * docs: add TODO comment about creating two environments * fix(test/deploy-lts): reduce sharedCacheDuration * refactor: make test settings compose files and env files explicit, no defaults chore: remove deploy test feat: add lts tests * fix(test/repo_client-lts): use wikibase-lts for client * fix: remove interwiki links test from quickstatements test * fix: reference deploy-lts correctly * fix: only include wikibase/build.env in repo/repo-lts tests * fix: deploy-lts wikibase build env * docs: rewrite changelogs * fix(nx): we do not use workspace changelogs * docs: remove old CHANGES file * docs: release checklist update * chore: remove todo * docs: deploy updating * refactor: DRY test settings * chore: prepare wikibase-lts and deploy-lts release bump version package.json already, so just the git tag is missing cleaned changelog files, so they are ready to go as well * test: add missing docker-compose to repo_client-lts * fix: makeTestSettings later, should fix results dir * fix: build.env path for repo-lts test --------- Co-authored-by: Loren Johnson <lorenjohnson@gmail.com>
1 parent c6056a0 commit b5fa284

File tree

73 files changed

+1770
-474
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

73 files changed

+1770
-474
lines changed

.github/actions/pull-ghcr/action.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ runs:
1111
1212
images=(
1313
"wikibase"
14+
"wikibase-lts"
1415
"elasticsearch"
1516
"wdqs"
1617
"wdqs-frontend"

.github/workflows/_build_test.yml

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ jobs:
2424
matrix:
2525
imageName:
2626
- wikibase
27+
- wikibase-lts
2728
- elasticsearch
2829
- wdqs
2930
- wdqs-frontend
@@ -50,12 +51,17 @@ jobs:
5051
matrix:
5152
suite:
5253
- repo
54+
- repo-lts
5355
- fedprops
56+
- fedprops-lts
5457
- repo_client
58+
- repo_client-lts
5559
- quickstatements
60+
- quickstatements-lts
5661
- pingback
62+
- pingback-lts
5763
- elasticsearch
58-
- deploy
64+
- elasticsearch-lts
5965

6066
steps:
6167
- uses: actions/checkout@v4
@@ -89,4 +95,4 @@ jobs:
8995
- lint
9096
- test
9197
steps:
92-
- run: true
98+
- run: "true"

.github/workflows/build_publish_image_release.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ on:
99
- 'wdqs-frontend@*'
1010
- 'wdqs-proxy@*'
1111
- 'wikibase@*'
12+
- 'wikibase-lts@*'
1213

1314
permissions:
1415
contents: write

.github/workflows/create_release.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,14 @@ on:
1010
default: 'wikibase'
1111
options:
1212
- deploy
13+
- deploy-lts
1314
- elasticsearch
1415
- quickstatements
1516
- wdqs
1617
- wdqs-frontend
1718
- wdqs-proxy
1819
- wikibase
20+
- wikibase-lts
1921
- All projects with unreleased changes
2022
dry_run:
2123
description: "Dry run, don't do it yet."

CHANGES.md

Lines changed: 0 additions & 303 deletions
This file was deleted.

README.md

Lines changed: 11 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -133,25 +133,23 @@ For more information on testing, see the [README](./test/README.md).
133133

134134
## Release process
135135

136-
WBS Deploy and WBS images are released using this repository. The process involves updating all upstream component versions to be used, building images, testing all the images together and finally publishing them.
136+
WBS Deploy and WBS images are released using this repository. The process
137+
involves updating upstream component versions to be used, building images,
138+
testing all the images together and finally publishing them.
137139

138140
### Release checklist Phabricator template
139141

140142
```
141143
- [ ] **Pending issues as subtasks**. If any open tickets need to be resolved and/or related changes need to be included in the release, add them as subtasks of this release ticket.
142-
- [ ] **To release breaking changes** as a new major version of WBS Deploy, create a new branch called `deploy-X`, where `X` is the new major version.
143-
- [ ] **Create a release PR** from a release preparation branch with the following changes targeting the appropriate `deploy-X` release branch.
144-
- [ ] **Backport from `main`** by cherrypicking commits from `main` to the release preparation branch.
145-
- [ ] **Update `build/*/build.env`** files by adjusting WBS versions and upstream versions. You can find further instructions in the `build.env` files themselves.
146-
- [ ] **Update `CHANGES.md`** by adding a section following the example of previous releases.
147-
- [ ] **CI should be green**. Tests may need adjustments in order to pass for the new version. Minor releases are likely to pass without any adjustments. Try re-running tests on failure, some specs could be flaky.
148-
- [ ] **Do a sanity check by manually reviewing a running instance using your build**. This can be done locally on your machine or on a public server. You can find built images from your release preparation branch on the [GitHub Container Registry](https://github.com/wmde/wikibase-release-pipeline/pkgs/container/wikibase%2Fwikibase) tagged with `dev-BRANCHNAME`, e.g., `dev-releaseprep`. This tag can be used to set up an instance running your release preparation version.
149-
- [ ] **Get two reviews on the release PR** so that it is ready to be merged. **Merging to `deploy-X` later will trigger the release to Docker Hub.** Do not merge yet!
150-
- [ ] **Prepare communication** by creating a [release announcement](https://drive.google.com/drive/folders/1iZMbdXGPsG0pLs-_HrniT5ac28aw1Edu) using a template.
144+
- [ ] **Update `build/*/build.env`** bump upstream versions by changing `build.env` files. You can find further instructions in the `build.env` files themselves.
145+
- [ ] **Merge all pending changes to `main`** releases are always done from the `main` branch.
146+
- [ ] **Verify CI on `main` is green**
147+
- [ ] **Do a sanity check by manually reviewing a running instance using your build**. This can be done locally on your machine or on a public server. You can find built images from your release preparation branch on the [GitHub Container Registry](https://github.com/wmde/wikibase-release-pipeline/pkgs/container/wikibase%2Fwikibase) tagged with `dev-GITHUB_ACTIONS_RUN_NUMBER`, e.g., `dev-123456789`. This tag can be used to set up an instance running your release preparation version.
148+
- [ ] **Release Dry Run** by running the Github Action [Create a WBS Release](https://github.com/wmde/wikibase-release-pipeline/actions/workflows/create_release.yml) in Dry Run mode. Check the output and review version and changelog generated.
149+
- [ ] **Prepare communication** by creating a [release announcement](https://drive.google.com/drive/folders/1iZMbdXGPsG0pLs-_HrniT5ac28aw1Edu).
151150
- [ ] **Coordinate with ComCom on timing the publication of the release**. Talk to SCoT (ComCom, technical writer) about this.
152-
- [ ] **Publish the release** by merging the release branch into the `deploy-X` branch. **ATTENTION: This will automatically push images to Docker Hub!**
153-
- [ ] **Update Dockerhub README** (from `./build/*/dockerhub.md`) for major versions, to make each image readme contain a link to appropriate version docs.
154-
- [ ] **Merge back to main in a separate PR** from `deploy-X` to have adjustments to `CHANGES.md` and the like available on `main` too. Changes from `variables.env` should only be taken from a release of the latest version so that `main` always references the build of the latest components.
151+
- [ ] **Publish the release** by running the Github Action [Create a WBS Release](https://github.com/wmde/wikibase-release-pipeline/actions/workflows/create_release.yml) **ATTENTION: This will automatically push images to Docker Hub!**
152+
- [ ] **Update Dockerhub README** (from `./build/*/dockerhub.md`) if required (e.g. major versions changed).
155153
156154
You`re done. **Congratulations!**
157155
```

build/build.sh

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,17 +71,22 @@ else
7171
)
7272
fi
7373

74-
# transform TAGS to build args
7574
IMAGE_NAME=$(jq -r '.name' package.json)
7675

7776
# publish to Dockerhub
7877
if [ "$PUBLISH" == true ]; then
7978
# IMAGE_REGISTRY implies dockerhub if empty
8079
IMAGE_NAMESPACE=wikibase
80+
81+
# Removes "-lts" from end of name if it exists.
82+
# LTS products are published on Dockerhub on the main image name
83+
IMAGE_NAME=${IMAGE_NAME%-lts}
84+
8185
# build/test in CI
8286
elif [ "$GITHUB_ACTIONS" == true ]; then
8387
IMAGE_REGISTRY=ghcr.io
8488
IMAGE_NAMESPACE="${GITHUB_REPOSITORY_OWNER}/wikibase"
89+
8590
# local build
8691
else
8792
IMAGE_NAMESPACE=wikibase

build/wikibase-lts/CHANGELOG.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
## 1.0.1 (2025-01-XX)
2+
3+
4+
### 🏡 Chore
5+
6+
- Bump MediaWiki to 1.39.10
7+
- Bump extensions
8+
- Removes re-installation of extensions already packaged with MediaWiki (ConfirmEdit, Nuke, Scribunto, SyntaxHighlight_Geshi, VisualEditor)
9+
10+
### 📖 Documentation
11+
12+
- Link to MediaWiki bundled extensions
13+
- Switch from `.example.com` to `.example`
14+

build/wikibase-lts/Dockerfile

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
# Build configuration
2+
ARG PHP_IMAGE_URL
3+
ARG COMPOSER_IMAGE_URL
4+
5+
# ###########################################################################
6+
# Based on https://github.com/wikimedia/mediawiki-docker/blob/1161796f04d6a6bcbec9fb4c67a8ce7248392403/1.41/apache/Dockerfile
7+
# hadolint ignore=DL3006
8+
FROM ${PHP_IMAGE_URL} as mediawiki
9+
10+
SHELL ["/bin/bash", "-exu", "-c"]
11+
12+
# System dependencies
13+
RUN apt-get update; \
14+
apt-get install -y --no-install-recommends \
15+
git \
16+
gettext-base \
17+
librsvg2-bin \
18+
imagemagick \
19+
# Required for SyntaxHighlighting
20+
python3 \
21+
# Required for Scribunto
22+
lua5.1 \
23+
; \
24+
rm -rf /var/lib/apt/lists/*
25+
26+
# Install the PHP extensions we need
27+
# hadolint ignore=DL4006
28+
RUN savedAptMark="$(apt-mark showmanual)"; \
29+
\
30+
apt-get update; \
31+
apt-get install -y --no-install-recommends \
32+
libicu-dev \
33+
libonig-dev \
34+
libbz2-dev=1.* \
35+
; \
36+
\
37+
docker-php-ext-install -j "$(nproc)" \
38+
bz2 \
39+
calendar \
40+
intl \
41+
mbstring \
42+
mysqli \
43+
opcache \
44+
; \
45+
\
46+
pecl install APCu-5.1.21; \
47+
docker-php-ext-enable \
48+
apcu \
49+
; \
50+
rm -r /tmp/pear; \
51+
\
52+
# reset apt-mark's "manual" list so that "purge --auto-remove" will remove all build dependencies
53+
apt-mark auto '.*' > /dev/null; \
54+
apt-mark manual $savedAptMark; \
55+
ldd "$(php -r 'echo ini_get("extension_dir");')"/*.so \
56+
| awk '/=>/ { print $3 }' \
57+
| sort -u \
58+
| xargs -r dpkg-query -S \
59+
| cut -d: -f1 \
60+
| sort -u \
61+
| xargs -rt apt-mark manual; \
62+
\
63+
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \
64+
rm -rf /var/lib/apt/lists/*
65+
66+
# Enable Short URLs
67+
RUN a2enmod rewrite; \
68+
{ \
69+
echo "<Directory /var/www/html>"; \
70+
echo " RewriteEngine On"; \
71+
echo " RewriteCond %{REQUEST_FILENAME} !-f"; \
72+
echo " RewriteCond %{REQUEST_FILENAME} !-d"; \
73+
echo " RewriteRule ^ %{DOCUMENT_ROOT}/index.php [L]"; \
74+
echo "</Directory>"; \
75+
} > "$APACHE_CONFDIR/conf-available/short-url.conf"; \
76+
a2enconf short-url
77+
78+
# Enable AllowEncodedSlashes for VisualEditor
79+
RUN sed -i "s/<\/VirtualHost>/\tAllowEncodedSlashes NoDecode\n<\/VirtualHost>/" "$APACHE_CONFDIR/sites-available/000-default.conf"
80+
81+
# set recommended PHP.ini settings
82+
# see https://secure.php.net/manual/en/opcache.installation.php
83+
RUN { \
84+
echo 'opcache.memory_consumption=128'; \
85+
echo 'opcache.interned_strings_buffer=8'; \
86+
echo 'opcache.max_accelerated_files=4000'; \
87+
echo 'opcache.revalidate_freq=60'; \
88+
} > /usr/local/etc/php/conf.d/opcache-recommended.ini
89+
90+
# MediaWiki setup
91+
WORKDIR /var/www/html
92+
ARG MEDIAWIKI_VERSION
93+
# hadolint ignore=DL4006
94+
RUN set -eux; \
95+
curl -fSL "https://releases.wikimedia.org/mediawiki/$(echo ${MEDIAWIKI_VERSION} | cut -d. -f1,2)/mediawiki-${MEDIAWIKI_VERSION}.tar.gz" -o mediawiki.tar.gz; \
96+
tar -x --strip-components=1 -f mediawiki.tar.gz; \
97+
rm -r mediawiki.tar.gz; \
98+
install -d /var/log/mediawiki -o www-data
99+
100+
# ###########################################################################
101+
# hadolint ignore=DL3006
102+
FROM ${COMPOSER_IMAGE_URL} as composer
103+
104+
COPY --from=mediawiki --chown=nobody:nogroup /var/www/html /var/www/html
105+
WORKDIR /var/www/html
106+
107+
COPY composer.local.json composer.local.json
108+
109+
# WORKAROUND for https://phabricator.wikimedia.org/T372458
110+
# Take wikibase submodules from github as phabricator rate limits us
111+
COPY --chown=nobody:nogroup --chmod=755 \
112+
wikibase-submodules-from-github-instead-of-phabricator.patch \
113+
/tmp/wikibase-submodules-from-github-instead-of-phabricator.patch
114+
USER root
115+
RUN apt-get update; \
116+
apt-get install -y --no-install-recommends \
117+
patch \
118+
; \
119+
rm -rf /var/lib/apt/lists/*
120+
USER nobody
121+
122+
ARG WIKIBASE_COMMIT
123+
SHELL ["/bin/bash", "-euo", "pipefail", "-c"]
124+
RUN set -x; \
125+
git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/Wikibase /var/www/html/extensions/Wikibase && \
126+
git -C /var/www/html/extensions/Wikibase checkout ${WIKIBASE_COMMIT} && \
127+
patch -d /var/www/html/extensions/Wikibase -Np1 </tmp/wikibase-submodules-from-github-instead-of-phabricator.patch && \
128+
rm /tmp/wikibase-submodules-from-github-instead-of-phabricator.patch && \
129+
git -C /var/www/html/extensions/Wikibase submodule update --init --recursive && \
130+
rm -f /var/www/html/extensions/Wikibase/.travis.yml && \
131+
find /var/www/html/extensions/Wikibase -name ".git*" -exec rm -rf {} +
132+
133+
ARG ALL_EXTENSIONS="Babel,cldr,CirrusSearch,Elastica,EntitySchema,OAuth,UniversalLanguageSelector,WikibaseCirrusSearch,WikibaseManifest"
134+
ARG BABEL_COMMIT
135+
ARG CLDR_COMMIT
136+
ARG CIRRUSSEARCH_COMMIT
137+
ARG ELASTICA_COMMIT
138+
ARG ENTITYSCHEMA_COMMIT
139+
ARG OAUTH_COMMIT
140+
ARG UNIVERSALLANGUAGESELECTOR_COMMIT
141+
ARG WIKIBASECIRRUSSEARCH_COMMIT
142+
ARG WIKIBASEMANIFEST_COMMIT
143+
ARG WIKIBASEEDTF_COMMIT
144+
ARG WIKIBASELOCALMEDIA_COMMIT
145+
146+
SHELL ["/bin/bash", "-euo", "pipefail", "-c"]
147+
RUN set -x; \
148+
IFS=',' read -ra EXTENSIONS <<< ${ALL_EXTENSIONS}; \
149+
for EXTENSION in "${EXTENSIONS[@]}"; do \
150+
rm -rf "extensions/${EXTENSION}"; \
151+
git clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/${EXTENSION}" "extensions/${EXTENSION}"; \
152+
EXTENSION_COMMIT_VAR="${EXTENSION^^}_COMMIT"; \
153+
EXTENSION_COMMIT="${!EXTENSION_COMMIT_VAR}"; \
154+
git -C "extensions/${EXTENSION}" checkout "${EXTENSION_COMMIT}"; \
155+
git -C "extensions/${EXTENSION}" submodule update --init --recursive; \
156+
rm -rf "extensions/${EXTENSION}/.git*"; \
157+
find "extensions/${EXTENSION}" -name ".git*" -exec rm -rf {} +; \
158+
done; \
159+
\
160+
git clone "https://github.com/ProfessionalWiki/WikibaseEdtf.git" "extensions/WikibaseEdtf"; \
161+
git -C "extensions/WikibaseEdtf" checkout "${WIKIBASEEDTF_COMMIT}"; \
162+
rm -rf "extensions/WikibaseEdtf/.git*"; \
163+
\
164+
git clone "https://github.com/ProfessionalWiki/WikibaseLocalMedia.git" "extensions/WikibaseLocalMedia"; \
165+
git -C "extensions/WikibaseLocalMedia" checkout "${WIKIBASELOCALMEDIA_COMMIT}"; \
166+
rm -rf "extensions/WikibaseLocalMedia/.git*"; \
167+
\
168+
rm -rf vendor && \
169+
rm -rf composer.lock && \
170+
ls -la && pwd && \
171+
composer install --no-dev -vv -n
172+
173+
# ###########################################################################
174+
# hadolint ignore=DL3006
175+
FROM mediawiki AS wikibase
176+
177+
LABEL org.opencontainers.image.source="https://github.com/wmde/wikibase-release-pipeline"
178+
179+
ARG GIT_CURRENT_REVISION
180+
181+
# Set error_reporting PHP.ini settings
182+
# This is needed with PHP8+ and MediaWiki 1.39, as Wikibase contains deprecated code
183+
# TODO: remove this and see how far we get
184+
RUN { \
185+
echo 'error_reporting = E_ALL ^ E_DEPRECATED'; \
186+
} > /usr/local/etc/php/conf.d/error_reporting.ini
187+
188+
COPY --from=composer --chown=nobody:nogroup /var/www/html /var/www/html
189+
190+
# Remove world writable flag, in combination with sticky bit it breaks the w link
191+
# https://github.com/wmde/wikibase-release-pipeline/commit/545c7aeec8d0245dc597d500afc934b40e656b3c
192+
# Make upload path writable for the webserver user
193+
RUN chmod o-w /var/www/html && \
194+
ln -s /var/www/html/ /var/www/html/w && \
195+
chown www-data /var/www/html/images -R
196+
197+
COPY wikibase-php.ini /usr/local/etc/php/conf.d/wikibase-php.ini
198+
COPY entrypoint.sh /entrypoint.sh
199+
COPY jobrunner-entrypoint.sh /jobrunner-entrypoint.sh
200+
COPY htaccess /var/www/html/.htaccess
201+
COPY LocalSettings.d LocalSettings.d
202+
COPY default-extra-install.sh /default-extra-install.sh
203+
COPY oauth.ini /templates/oauth.ini
204+
COPY LocalSettings.wbs.php /templates/LocalSettings.wbs.php
205+
206+
ENV DB_NAME=my_wiki \
207+
MW_WG_SITENAME=wikibase \
208+
MW_WG_LANGUAGE_CODE=en \
209+
ELASTICSEARCH_PORT=9200
210+
211+
ENTRYPOINT ["/bin/bash"]
212+
CMD ["/entrypoint.sh"]
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
<?php
2+
3+
# Wikibase Repository
4+
wfLoadExtension( 'WikibaseRepository', "$IP/extensions/Wikibase/extension-repo.json" );
5+
require_once "$IP/extensions/Wikibase/repo/ExampleSettings.php";
6+
7+
# Wikibase Client
8+
wfLoadExtension( 'WikibaseClient', "$IP/extensions/Wikibase/extension-client.json" );
9+
require_once "$IP/extensions/Wikibase/client/ExampleSettings.php";

0 commit comments

Comments
 (0)