Commands for maintaining a language repository

This document contains instructions to guide you on managing your language repository, hence helping you translate Python’s documentation to your language.

See the Sample Workflows for an automated alternative to this guide.

These commands are written in Linux Shell, and should work in Windows Subsystem Linux (WSL), but feel free to use the same logic in other languages e.g. Python.

Note

Where you see ${LANGCODE} in the commands below, replace it with your language code (e.g. ‘uk’, ‘pt_BR’) or set the variable (e.g. LANGCODE=pt_BR) before running the commands.

Clone CPython repository

It is necessary to have a local clone of CPython’s source code repository in order to update translation files and to build translated documentation.

From inside your language repository, run:

BRANCH=3.14
git clone --depth 1 https://github.com/python/cpython --branch $BRANCH

Use --depth 1 to shallow clone, which avoids downloading all the 800 MB of data from CPython’s repository.

Optionally, you could also add --no-single-branch to git clone command which would make all branches available, allowing to switch between one branch and another. But this is not needed if you are working on the translation of a single branch.

Install requirements

Creating a virtual environment

Optionally, create a virtual environment (short: venv) to keep all Python package installations in it, then activate this venv to ensure commands are run from it:

python -m venv .venv
source .venv/bin/activate

Install Python packages

Update pip and then install the required packages:

python -m pip install --upgrade pip
python -m pip install sphinx-intl>=2.1.0 pomerge powrap sphinx-lint

Alternatively, put all Python packages inside a requirements.txt file and install it using pip’s -r flag.

Transifex CLI tool

Install the Transifex CLI client, required to interact with Transifex:

cd .venv/bin
curl -s -o- https://raw.githubusercontent.com/transifex/cli/master/install.sh
cd ../..

Note

The above commands result in the tx binary being downloaded into the .venv/bin directory. Feel free to install this tool wherever you want, but preferably in a directory already in the PATH so that tx works without its full path.

Updating the translations

For language teams that coordinate translation efforts on Transifex, updating translation means pulling the translation strings.

Recommendation

Before pulling translations, consider updating the .tx/config to have an up-to-date mapping of project/resources. For this, it is required to generate the documentation’s pot files (template of po files), so start with the pot. Alternatively, you can skip it and pull translations, but new translation resources in Transifex could be not mapped, and hence wouldn’t be pulled.

Generating pot files

Sphinx’s gettext builder can be used for generating pot files:

make -C cpython/Doc/ ALLSPHINXOPTS='-E -b gettext -D gettext_compact=0 -d build/.doctrees . locales/pot' build

There should now be a cpython/Doc/locales/pot/ directory containing all of the pot files.

Generating a .tx/config file

Once you have the pot files, make use of sphinx-intl to generate the .tx/config:

cd cpython/Doc/locales
sphinx-intl create-txconfig
sphinx-intl update-txconfig-resources \
    --transifex-organization-name python-doc \
    --transifex-project-name=python-newest \
    --locale-dir . --pot-dir pot

This step should take some time to finish. Once it is complete, there should be a cpython/Doc/locales/.tx/config file containing a list of resources based on the previously generated pot files.

As a final touch, we copy the .tx/config to the language repository making proper tweaks so one can download translations from Transifex or upload local translation changes, all this from repository’s root directory:

cd ../../..   # back to language repository root directory
mkdir -p .tx
sed cpython/Doc/locales/.tx/config \
    -e "s|^file_filter  = .*|&\nx&|;" \
    -e "s|^source_file  = pot/|source_file  = cpython/Doc/locales/pot/|" \
    > .tx/config
sed -i .tx/config \
    -e "s|^xfile_filter  = ./<lang>/LC_MESSAGES/|trans.${LANGCODE}  = |;"

Remapping translation and Transifex resources is complete.

Pulling translations

To download translations from Transifex using Transifex CLI tool:

tx pull -l ${LANGCODE} -t -f

Argument explanations:

  • -l ${LANGCODE} – specify the language code so that tx doesn’t pull all languages.

  • -t – specify that we want translations

  • -f – force pulling all files, because without this sometimes changes in Transifex are not downloaded

Wrapping the translation files

After pulling, it is highly recommended to use powrap on the po files to make them look better:

powrap --quiet *.po **/*.po

Alternatively, you can use --modified flag to save time and apply only to changed files.

Commit and push translation changes

The following commands are recommended for committing and pushing your translations to your language repository.

git diff -I'^"POT-Creation-Date: ' --numstat *.po **/*.po | cut -f3 | xargs -r git add
git add $(git ls-files -o --exclude-standard *.po **/*.po) .tx/config
git diff-index --quiet HEAD || { git commit -m "Update translations" && git push; }

It is not recommended to simply git add (stage) all PO files because this would also stage (and then commit) the translation files that have only irrelevant changes in their POT-Creation-Date header field (i.e. date when the PO was updated against the POT).

The first command first git-add modified tracked files that does not exclusively match changes in POT-Creation-Date header, hence relevant changes are included.

The second command will git-add untracked po files that may have been newly created on the latest ‘tx pull’ run. It also adds .tx/config file.

The last command will only commit and push if any file was git-added in the above commands.

Build translated documentation

Useful for testing the translations, spotting syntax errors and viewing the result of your contribution.

To build translated documentation, run:

cp --parents *.po **/*.po cpython/Doc/locales/${LANGCODE}/LC_MESSAGES/
make -C cpython/Doc venv
make -C cpython/Doc SPHINXOPTS="--keep-going -D gettext_compact=0 -D language=${LANGCODE}" html

The first command copies the translation files (.po) into cpython’s locale_dir, which is required for it to be recognized.

The second command creates a pre-configured virtual environment using the Makefile from CPython’s Doc directory.

Finally, build using the Makefile. Here is an explanation of the arguments used:

  • -C cpython/Doc – changes the current directory to run the make command

  • SPHINXOPTS – this variable should contain any CLI modifier command you want to pass

  • --keep-going – even if it fails, go all way to the end to bring up all errors

  • -D gettext_compact=0 – override sphinx settings to consider one PO file == one doc page

  • -D language=$LANGCODE – override sphinx settings to build in the desired $LANGCODE

  • html – the Makefile target that triggers the Sphinx’s html builder

Viewing the documentation in a web browser

Just build translated documentation and then open in a browser, no secrets. See below a one-line command to use your default web browser to open the index.html:

python -c "import os, webbrowser; webbrowser.open('file://cpython/Doc/build/html/index.html')"

Notice index.html can be replaced with any file, e.g. 'library/os.html'.

Linting the translation files

sphinx-lint is great to spot translation errors that will didn’t spot e.g. trailing whitespace in the string, reST directive not properly surrounded with whitespace, etc. It’s highly recommended.

sphinx-lint *.po **/*.po

Merging translations into another branch

This is useful when you want to replicate a translation from the CPython branch currently being translated to another older branch. E.g. 3.14 is currently being translated, but 3.13 has that same string and could make use of the contributed translations.

CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
TARGET_BRANCH=3.12
pomerge --from-files *.po **/*.po
git checkout ${TARGET_BRANCH}
pomerge --to-files *.po **/*.po

After the above command, the translation from the current branch were applied to the previous branch 3.13. Now, one can verify lines are wrapped:

powrap --modified *.po **/*.po

Done changing, let’s commit and push these changes, and go back the original branch:

git diff -I'^"POT-Creation-Date: ' --numstat *.po **/*.po | cut -f3 | xargs -r git add
git diff-index --quiet HEAD || { git commit -m "Merge translations into ${TARGET_BRANCH}" && git push; }
git checkout ${CURRENT_BRANCH}