Building Releases
=================

Roundup is a pure Python application with no binary components. This file
describes how to build a source release. To find out how to install
Roundup, read the doc/installation.txt file.

Roundup release checklist:

1.  Regenerate locale/roundup.pot.  See "Extracting Translatable
    Messages" in doc/developers.txt.  This is best done some time in
    advance of the release to allow for translators to update their
    translations. Merge into .po files by running 'make *.po'
    in the locale directory. Run:

       cd locale
       make merge
       cd ..

    'make merge' will remake template if the source and html files are
    out of date. 'make template' will regenerate roundup.pot. Touch
    any .py file in the roundup/ directory to force 'make template'
    to run.

2.  Run unit tests! They should pass successfully. Install pytest
    using pip2/pip3 for python2 and python3. Then invoke pytest
    using both python versions from the top of the roundup tree:

        python3 -m pytest test/
        python2 -m pytest test/

3.  Update version in:
      CHANGES.txt (set date for version as well) 
      roundup/__init__.py
      website/www/index.txt (current stable version, release highlights)
      website/www/conf.py  (update copyright, version auto-set from
          roundup/__init__.py)
      scripts/Docker/Dockerfile update value of
          org.opencontainers.image.version
3a. Update license end date in COPYING.txt
3b. Update doc/acknowledgements.txt (add section for
    release, churn contributers etc.). (Use hg churn -c -r ####..####)

4.  Update documentation
      doc/announcement.txt
      doc/upgrading.txt

    Use::

       sed -ne '/^20[0-9][0-9]-XX/,/^202[0-9]-/{/^-/p}' CHANGES.txt | wc -l

   to get number of features/fixes included in the release.

4a. cd to website/www and run 'make linkcheck'. See output in
    _tmp/linkcheck/output.txt
       fix broken references in docs
       verify redirects are correct

    Use::

        grep '\[broken\]\s*htt'  _tmp/linkcheck/output.txt

    to find external refs only. Internal refs will probably fail
    because they don't use :ref: or :doc: roles to reference them.
    They are referenced using html link syntax.

5.  Update setup.py info if needed (contacts, classifiers, etc.). When
    releasing check that Development Status matches release: stable,
    beta, alpha etc.

    Check that metadata is valid and long descriptions is proper reST:

      python3 setup.py check --restructuredtext --metadata --strict
   
6. Rebuild .mo translation files in distribution

      cd locale
      make
      cd ..

7.  Remove previous build files

      python3 setup.py clean --all
      rm -rf build/share  # deletes locale .mo files

    Clean out all *.orig, *.rej, .#* files from the source.

      find . -name '*.orig' -exec rm {} \;
      find . -name '*.rej' -exec rm {} \;
      find . -name '.#*' -exec rm {} \;

7a.

    Build including new .mo files built in 6.

      python3 setup.py build

    (sdist generation will fail if this isn't done)

8.  Rebuild documentation in "share/doc/roundup/html"

      cd doc
      make
      cd ..

    runs commands to turn man pages into html files and adds them to
    html_extra/man_pages subdir. Then it generates html text from
    running roundup_admin. Then it generates a current copy of a
    config.ini file. Then it runs:

       python3 setup.py build_doc

9.  Generate source distribution:

      python3 setup.py sdist

    (if you find sdist a little verbose, add "--quiet" to the end of the
     command)

9a. 2021/04/17 skip this for now. Need to make sure that whl installs
    executable scripts properly and update these directions to test.

       python3 setup.py bdist_wheel

    to create binary distributions in wheel format. (egg format is
    deprecated.)

10. Check the roundup.egg-info/SOURCES.txt to make sure that any new files are
    included.  (use hg status --rev <last release or tag>:tip to list changed
    added and removed files. Last release e.g. 1.5.1 where tip is what would
    become 1.6) E.G.
      
      hg status --rev 2.2.0:tip | sed -ne 's/^A //p' | while read i ; \
      do echo $i; grep "$i" roundup.egg-info/SOURCES.txt; done | \
      uniq -c | sort -rn

    Anything with a count of 1 is not in the manifest.
    If there are missing files that should be in the manifest,
    edit MANIFEST.in to include them. For format docs see
    https://packaging.python.org/guides/using-manifest-in/#using-manifest-in
    (Note: files under website/ shouldn't be in the manifest.)
10a: Check for removed files still in manifest:

      hg status --rev 2.2.0:tip | sed -ne 's/^R //p' | while read i ; \
      do echo $i; grep "$i" roundup.egg-info/SOURCES.txt; done | \
      uniq -c | sort -n

    any file with a count of 2 or more needs to be removed from
    MANIFEST.in and possibly cleaned out of the build tree.

10b: if you added/removed files rebuild starting at step 6a.

11. Unpack the new tarball created in dist/roundup-<version>.tar.gz
    file in /tmp then
      a) run tests using installed pytest run under
         python3. (python3 -m pytest test/)
      b) demo.py
    with all available Python 3 versions.
11a. (TBD how to test wheel binary distribution before uploading.)

11b. Generate GPG signature file

       cd dist
       gpg --detach-sign --armor -u 1F2DD0CB756A76D8 <filename>.tar.gz

     you should be prompted to use the roundup release key. If not you
     can add  --local=roundup-devel@lists.sourceforge.net.
     This will create a file by the name <filename>.tar.gz.asc.

     Move file to website/www/signatures directory

       mv <filename>.tar.gz.asc ../website/www/signatures/.
       hg add ../website/www/signatures/<filename>.tar.gz.asc
       # commiting the file will be done in step 12
       cd ..

    Add a link to the signature to doc/security.txt. Add a new link
    to the start of the signature list in doc/security.txt (look for
    the word multicol).

12. Assuming all is well commit and tag the release in the version-control
    system.
      a) hg commit ... # commit any edits from steps 1-5
      b) hg tag 2.1.0  # use right version. Should create/commit a changeset
      c) hg push       # update main repo
      d) hg sum        # verify that the tag shows up

13. Upload source distribution to PyPI - requires you sign up for a
    pypi account and be added as a maintainer to roundup. Ask existing
    maintainer for access. Do this using twine (pip install twine).

    The original directions used twine to upload the tarball and the
    signature, but as of May 2023, PyPI no longer accepts signature
    files. So we publish the signature as part of the website.

    Use twine to upload the distribution tarball. E.G.

       twine upload --repository pypi <filename>.tar.gz

    The distribution file should appear on
    https://pypi.python.org/pypi/roundup in no time. If you are using
    python older than 2.7.13 you need a .pypirc shown below since the
    URL has changed.

    You can also use twine to upload the .whl (wheel) format
    distributions (if created). Follow the directions for generating
    the gpg asc files and place the .whl.asc in the signature
    directory.

    Another way to upload is to use:

      python3 setup.py sdist upload --repository pypi

    BUT this rebuilds the source distribution tarball and uploads it.
    This means that you have uploaded something that is not tested.
    Also the metadata in the file changes and will not match the GPG
    signature you commited in step 12. So use twine.

14. Refresh website.
      website/README.txt
      https://www.roundup-tracker.org/ should state that the stable
      version is the one that you released.
      https://www.roundup-tracker.org/docs.html should also match the
      released version (or at least the major non pre-release
      1.x/2.x version).

15. Send doc/announcement.txt to python-announce@python.org,
    roundup-users@lists.sourceforge.net,
    roundup-devel@lists.sourceforge.net, and lwn@lwn.net.
15b. Update entry on https://freshcode.club/projects/roundup-tracker
15c. Update entries for fossies by emailing announcement to
     announce@fossies.org
15d. Update entry on https://directory.fsf.org/wiki/Roundup.

16. Change the version in the URL that generates the "commits since
    release ...." badge. Also update Python test versions in the
    GitHub Actions section at:
      https://wiki.roundup-tracker.org/CiTestingEnvironment
16b. Update release info on wikipedia:
     https://en.wikipedia.org/wiki/Roundup_(issue_tracker)

     https://en.wikipedia.org/wiki/Comparison_of_issue-tracking_systems

17 Push release docker image to dockerhub
17a. install docker
17b. run: (issues, how to release a version e.g. to update alpine for
           security issues. Currently thinking that release tag is
	   rounduptracker/roundup:2.4.0-1, -2 etc. Then add a tag
	   rounduptracker/roundup:2.4.0 that moves to always tag
	   the latest -N release. Also roundup:latest points to the
	   newest -N for the newest roundup version.)

        docker build -t rounduptracker/roundup:2.4.0 \
           --build-arg="source=pypi" -f scripts/Docker/Dockerfile .

     to create the docker image. *Change 2.4.0 to current version*
     Always use the exact release tag.
17c. vulnerability scan local image using:

        docker run --rm --volume \
	/var/run/docker.sock:/var/run/docker.sock \
            --name Grype anchore/grype:latest rounduptracker/roundup:2.4.0

     should report no vulnerabilities (note match version with current
     build)

     Also can scan (optionally) using trivy:

        docker run -it --rm --volume \
	/var/run/docker.sock:/var/run/docker.sock \
          --name trivy aquasec/trivy:latest image --exit-code 1 \
          --ignore-unfixed rounduptracker/roundup:2.4.0

     You may need to explicitly update/refresh the scanners with:
     "docker pull anchore/grype:latest" and similarly for
     aquasec/trivy if used.

     Note that some security issues may show up. If they are in the
     underlying OS package we can't do anything but make sure the
     latest python:3-alpine package is used to build the image. Look
     at the index digest on the image release page and compare it to
     the sha256 at the top of the Dockerfile.

17d. test roundup in demo mode:

        docker run -it --rm -p 8917:8080 \
         -v $PWD/tracker:/usr/src/app/tracker \
         rounduptracker/roundup:2.4.0 demo

     FIXME: right now the external port number (8917) in the above
       command is hardcoded in DOCKER. It can be overridden usng
       PORT_8080=9017 for example. However the host is always
       localhost. Consider replacing PORT_8080 with ORIGIN="host:port"
       or ORIGIN="port" so that the web URL can be made correct when
       running docker on a remote server.

17e. push to DockerHub login (login using 'docker login <username>'
     first and user must be member of rounduptracker org with ability
     to publish). Replace -N with the release number (e.g. -1, -2, -3...)

	 docker tag rounduptracker/roundup:2.4.0 roundup-tracker/roundup:latest
	 docker tag rounduptracker/roundup:2.4.0 roundup-tracker/roundup:2.4.0-N
         docker push rounduptracker/roundup:2.4.0
         docker push rounduptracker/roundup:2.4.0-N
         docker push rounduptracker/roundup  # update roundup:latest

-------------

If you get errors on the upload operation, you may need the following
~/.pypirc file as well

========
[distutils]
index-servers =
    test
    pypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: <your username on pypi.org here>
password: <your password here>

[test]
repository: https://test.pypi.org/legacy/
username: <your username on test.pypi.org here>
password: <your password here>
========

-------------

==========================
GPG public key operations.
==========================

LIST IN KEYRING
===============

$ gpg --list-keys -a roundup-devel@lists.sourceforge.net

   pub   rsa4096 2018-07-11 [SC] [expires: 2028-07-17]
         411E354B5D1AF26125D621221F2DD0CB756A76D8
   uid           [ultimate] Roundup Team (signing key for roundup
         releases) <roundup-devel@lists.sourceforge.net>
   sub   rsa4096 2018-07-11 [E] [expires: 2028-07-17]

EXTEND EXPIRATION DATE
======================

Needs private key and passphrse for private key

$ gpg --edit-key 411E354B5D1AF26125D621221F2DD0CB756A76D8

  > expire
  [add some number of months/years to it]
  > key 1
    [ this chooses the subkey "sub" ]
  > expire
  [add some number of months/years to the sub key ]
  > save
  [ saves both keys, will need the private key and passphrase ]

EXPORT NEW PUBLIC KEY
=====================

$ gpg --export -a roundup-devel@lists.sourceforge.net >> \
   tools/roundup.public.pgp.key

then edit roundup.public.pgp.key keeping only the last key that starts
with: -----BEGIN PGP PUBLIC KEY BLOCK-----

and add back the preamble that describes where to find doc for
it. Commmit new key to mercurial.

SAVE TO KEYSERVER
=================

$ gpg --keyserver pgp.mit.edu --send-keys \
     411E354B5D1AF26125D621221F2DD0CB756A76D8

update it on the keysserver so users can download it.