Mercurial > p > roundup > code
changeset 7340:7b9bddda9d2d
Add support for demo mode in docker.
roundup/demo.py
Make changes to allow exposed port in docker to be specified
separately from the port that demo mode binds to. Also permit
bind address specification as well.
roundup/scripts/roundup_demo.py:
Update required by changes in demo.py. Also move away from
positional arguments to prefer flag arguments. Required for
passing port and host specification. Flake8 fixes.
share/man/man1/roundup-demo.1
Document use of option flags rather than positional
params. Other cleanups.
doc/installation.txt:
Document new docker modes: demo, shell and admin.
Update docs:
overview section - reorg, added template info
for the impatient section - added docker demo mode reference,
more docs on top level demo.py use.
new section on docker demo mode
removed getting roundup section. folded into installing roundup.
also prior for the impatient section describes how to download.
install via pip in venv recommended supported method
document all provided templates. not just minimal and classic.
added index references.
move sections around, decreased sectin depth, reformatting
scripts/Docker/roundup_healthcheck:
When running roundup-demo, there is no tracker spec. So default to
demo if no tracker=directory args found. Prevent's docker from
reporting an unhealthy container when running demo.
scripts/Docker/roundup_start:
implement demo, shell, admin docker modes.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Sun, 14 May 2023 09:43:53 -0400 |
| parents | 5eadba24e148 |
| children | 7321c0e6c53e |
| files | CHANGES.txt doc/installation.txt roundup/demo.py roundup/scripts/roundup_demo.py scripts/Docker/roundup_healthcheck scripts/Docker/roundup_start share/man/man1/roundup-demo.1 |
| diffstat | 7 files changed, 1066 insertions(+), 444 deletions(-) [+] |
line wrap: on
line diff
--- a/CHANGES.txt Sun May 14 01:23:36 2023 -0400 +++ b/CHANGES.txt Sun May 14 09:43:53 2023 -0400 @@ -92,6 +92,10 @@ directory. This better identifies the directory when copied into the Zope framework. It also matches existing documentation. (John Rouilard) +- Dockerfile supports demo mode for instant gratification + 8-). Also supports shell and admin mode (John Rouillard) +- Multiple fixes/updates for installation documentation. + Including docker shell/admin/demo mdoes. (John Rouillard) Features:
--- a/doc/installation.txt Sun May 14 01:23:36 2023 -0400 +++ b/doc/installation.txt Sun May 14 09:43:53 2023 -0400 @@ -19,31 +19,59 @@ Overview ======== -Broken out separately, there are several conceptual pieces to a -Roundup installation: +A Roundup installation is made up of several pieces. + +Roundup scripts + These include the Roundup HTTP server, email gateway, administration + command-line interface, demo installer etc. These are usually placed + in a directory that is on your path. + +Roundup core code + Is installed into your Python's lib directory. We recommend + using a virtual environment for your Roundup installation. Roundup trackers - Trackers consist of issues (be they bug reports or otherwise), tracker - configuration file(s), web HTML files etc. Roundup trackers are initialised - with a "Template" which defines the fields usable/assignable on a - per-issue basis. Descriptions of the provided templates are given in - `choosing your template`_. - -Roundup support code - Installed into your Python install's lib directory. - -Roundup scripts - These include the email gateway, the roundup - HTTP server, the roundup administration command-line interface, etc. + Trackers consist of issues (be they bug reports or otherwise). Each tracker + is put in its own directory (called a tracker home) and has its own: + + * configuration files, + * HTML (web) files, + * database, + * logic files (detectors, schema, ...) + + Roundup trackers are initialised with a "template" which defines the + fields usable/assignable on a per-issue basis. Descriptions of the + provided templates are given in `choosing your template`_. Usually + you start with a template then modify the tracker to implement your + desired workflow. One Roundup instalation can support multiple + trackers with different look/feel and workflow. For The Really Impatient ======================== -If you just want to give Roundup a whirl Right Now, follow these -directions to download, unpack and run ``demo.py``. (Replace -``-2.2.0`` with the version number of the file you downloaded. On -systems that don't provide a ``python3`` program you can run -``python demo.py``.) +If you just want to give Roundup a whirl **Right Now**, follow these +directions to run ``demo.py``. Demo mode starts the `classic tracker`_ +without installing Roundup on your system. If you have Docker +installed, you can run `demo mode using docker`_ instead. + +This is also a way to spin up a development environment or even +deploy a tracker for a handful of users. + +You can choose different templates and backend databases using demo +mode. For example replacing ``demo.py`` (or ``demo`` if you are using +docker) with:: + + demo jinja2 anydbm + +will start the tracker using the jinja2 template with the dbm database +backend (rather then the default sqlite). See `Choosing Your +Template`_ for a description of available templates. + +(In the directions below, replace ``-2.2.0`` with the version number +of the file you downloaded. On systems that don't have a ``python3`` +program you can run ``python demo.py`` instead.) + +.. _install the source: 1. ``python3 -m pip download roundup`` 2. ``tar -xzvf roundup-2.2.0.tar.gz`` @@ -53,9 +81,10 @@ 3. ``cd roundup-2.2.0`` 4. ``python3 demo.py`` -This will set up a simple demo tracker on your machine. [1]_ -When it's done, it'll print out a URL to point your web browser at -so you may start playing. Three users will be set up: +This will set up a classic demo tracker on your machine without +installing Roundup. [1]_ When it's done, it'll print out a URL for +your web browser at so you can explore a Roundup tracker. Three users +are set up: 1. anonymous - the "default" user with permission to do very little 2. demo (password "demo") - a normal user who may create issues @@ -64,11 +93,11 @@ Note the demo tracker removes the detector (nosyreaction.py) that sends email notifications. If you later convert your demo tracker to -production you will need to copy in the detector to send notification +production you will need to replace the detector to send notification emails. -If you install Roundup, the ``demo.py`` script is available as -``roundup-demo``. +Once you install Roundup, you can use the ``roundup-demo`` command to +install new demo trackers. .. [1] Demo tracker is set up to be accessed by localhost browser. If you run demo on a server host, please stop the demo (using @@ -77,13 +106,78 @@ the ``web`` option in section ``[tracker]``, save the file, then re-run the demo.py program. +.. _demo mode using docker: + +Running in Demo Mode with Docker +-------------------------------- + +You can either: + +* use a published container from hub.docker.com with + ``rounduptracker/roundup:latest`` + +or + +* Use steps 1-3 to `install the source`_ then + +* build a local docker container using:: + + docker build -t roundup-app -f scripts/Docker/Dockerfile . + + (see `Docker Support`_ and `Building a Docker Container`_ for more + details) + +Start demo mode with [2]_:: + + docker run --rm -p 127.0.0.1:8917:8080 --name roundup_demo -v \ + $PWD:/usr/src/app/tracker rounduptracker/roundup:latest demo + +.. [2] Replace ``rounduptracker/roundup:latest`` with + ``roundup-app:latest`` if you built your own docker image. + +This will create a ``demo`` subdirectory which is your tracker's +home. It will also print the URL for exploring your new tracker. + +.. caution:: + Removing ``127.0.0.1:`` will make the tracker accessible from any + host with network access to your system. However the URL's created by + Roundup will still reference ``localhost`` unless you modify the + ``web`` url in the ``tracker`` section of ``config.ini`` and restart + the container [1]_. + +In the docker run command we used port 8917 for Roundup. When starting +Roundup, Docker may report a long error ending with: `bind: address +already in use.` This means that port 8917 is in use. When running +inside a Docker container, demo mode is unable to automatically find a +free port. You have to provide an unused port to ``-p``. + +To fix this, you can change the change the port mapping provided +with ``-p``. If you do this you **must** set the docker PORT_8080 +environment variable on the command line to match. (If Docker +ever fixes https://github.com/moby/moby/issues/3778 we won't need +to worry about this.) + +For example:: + + docker run --rm -e PORT_8080=9090 -p 127.0.0.1:9090:8080 -v \ + --name roundup_demo $PWD:/usr/src/app/tracker \ + rounduptracker/roundup:latest demo + +will run Roundup on port 9090 and Roundup will generate the correct +URL. + +To shut down the tracker and get your shell back, use control-c. You +can remove the tracker using ``rm -f`` on the ``demo`` directory. + Prerequisites ============= -Roundup requires Python 2.7 or 3.4 or newer with a functioning -anydbm module. Download the latest version from https://www.python.org/. -It is highly recommended that users install the latest patch version -of python as these contain many fixes to serious bugs. +Roundup requires Python 2.7 or 3.4 or newer with a functioning anydbm +or sqlite module. The version installed by most vendors should work if +it meets the version requirements. If necessary, you can download the +latest version from https://www.python.org/. It is highly recommended +that users install the latest patch version of Python as these contain +many fixes to serious bugs. Some variants of Linux will need an additional "python dev" package installed for Roundup installation to work. Debian and derivatives, are @@ -94,18 +188,18 @@ You may optionally install and use: +An RDBMS + Sqlite, MySQL and Postgresql are all supported by Roundup and will be + used if available. One of these is recommended if you are anticipating a + large user base (see `choosing your backend`_ below). Sqlite should + always be available. + Timezone Definitions Full timezone support requires pytz_ module (version 2005i or later) which brings the `Olson tz database`_ into Python. If pytz_ is not installed, timezones may be specified as numeric hour offsets only. This is optional but strongly suggested. -An RDBMS - Sqlite, MySQL and Postgresql are all supported by Roundup and will be - used if available. One of these is recommended if you are anticipating a - large user base (see `choosing your backend`_ below). Sqlite should - always be available. - Xapian full-text indexer The Xapian_ full-text indexer is also supported and will be used by default if it is available. This is strongly recommended if you are @@ -119,8 +213,8 @@ Note that capitalization is not preserved by the Xapian search. This is required to make the porter stemmer work so that searching - for silent also returns documents with the word silently. Note that - the current stemming implementation is designed for English. + for ``silent`` also returns documents with the word ``silently``. + Note that the current stemming implementation is designed for English. Whoosh full-text indexer The Whoosh_ full-text indexer is also supported and will be used by @@ -134,7 +228,7 @@ command if the tracker has existing data. Roundup was tested with Whoosh 2.5.7, but earlier versions in the - 2.0 series may work. Whoosh is a pure python indexer so it is slower + 2.0 series may work. Whoosh is a pure Python indexer so it is slower than Xapian, but should be useful for moderately sized trackers. It uses the StandardAnalyzer which is suited for Western languages. @@ -155,11 +249,6 @@ its TEMPLATE-INFO.txt file) you need to have the jinja2_ template engine installed. -pyjwt - To use jwt tokens for login (experimental), install `pyjwt`_ - (v1.7.1, v2.0.1 tested). If you don't have it installed, jwt - tokens are not supported. - docutils To use ReStructuredText rendering you need to have the `docutils`_ package installed. @@ -183,249 +272,134 @@ `Using Redis for Session Databases`_ in the `administration guide`_ for details. +pyjwt + To use JWT (JSON web tokens) for login (experimental), install `pyjwt`_ + (v1.7.1, v2.0.1 tested). If you don't have it installed, JWT's + are not supported. + Windows Service You can run Roundup as a Windows service if pywin32_ is installed. Otherwise it must be started manually. requests If you are using OAuth authentication with the roundup-mailgw - mail gateway you need to install the requests_ library. + mail gateway you must install the requests_ library. .. _Using Redis for Session Databases: admin_guide.html#using-redis-for-session-databases -Getting Roundup -=============== - -.. note:: - Some systems, such as Gentoo and NetBSD, already have Roundup - installed. Try running the command "roundup-admin -v". - If it runs and reports a current version, you may skip the - `Basic Installation Steps`_ below and go straight to +Installing Roundup +================== + +To get a production installation running will take 15-30 minutes. If +you want to spend less than 5 minutes to test Roundup without +installing it, see `For The Really Impatient`_. + +.. note:: Some systems, such as Gentoo and NetBSD, already have + Roundup installed. Try running the command "roundup-admin + -v". If it runs and reports the current version, you may + skip the `Standard installation`_ below and go straight to `configuring your first tracker`_. However it may be an old version. If so you should probably install it in a virtual environment from the Roundup web site or pypi. -Download the latest version from https://www.roundup-tracker.org/. - -Installation -============ - -Set aside 15-30 minutes. There's several steps to follow in your -installation: - -1. `basic installation steps`_ if Roundup is not installed on your system -2. `configuring your first tracker`_ that all installers must follow -3. then optionally `configure a web interface`_ -4. and optionally `configure an email interface`_ -5. `UNIX environment steps`_ to take if you're installing on a shared - UNIX machine and want to restrict local access to Roundup - -For information about how Roundup installs, see the `administration -guide`_. - -The following assumes that you are using the source distribution. - -Basic Installation Steps ------------------------- +If Roundup is not installed on your system, or needs to be updated, +there are multiple ways to install Roundup. + +* `Standard installation`_ using pip in a Virtual Environment is the + recommended standard. +* `Installing from downloaded source`_ allows more control over how + things are installed (including overwriting a vendor install). But + it also increases complexity as well. +* Use a prebuilt docker container from + ``rounduptracker/roundup:latest`` and follow the steps in + `Running Your Container`_. +* Install in a docker container by downloading the source + and following the steps in `Docker Support`_. + +There are several steps to get Roundup serving a tracker: + +1. Install using one of the methods listed above. +2. Configure your tracker following `configuring your first tracker`_ + for all install methods. +3. Optionally `configure a web interface`_. +4. Optionally `configure an email interface`_. +5. Follow `UNIX environment steps`_ to restrict local access to + Roundup if you're installing on a shared UNIX system. + +For information about what Roundup installs, see the +`What does Roundup install`_ section in the `administration guide`_. + + +Standard Installation +---------------------- Installation of Roundup using Python3 in a virtual environment is recommended. Use:: python3 -m venv /path/to/environment/roundup -then proceed as below after activating (assuming a Bourne like shell) -the Python environment using:: +Activate the Python environment (assuming a Bourne like shell) +using:: . /path/to/environment/roundup/bin/activate -You can use the alias ``deactivate`` to return to the normal Python -environment. If you create the virtual envirnment as a non-root user, -you can install below using the same user. - -To install the Roundup support code into your Python tree and Roundup -scripts into /usr/bin (substitute that path for whatever is -appropriate on your system). You need to have write permissions for -these locations, so you may need to run wthese commands with ``sudo`` -if root permission is required:: - - python setup.py install +To install the released Roundup core code into your Python tree and +Roundup scripts into ``/path/to/environment/roundup/usr/bin`` run:: + + python3 -m pip install roundup + +If everything went well, you should now be able to run:: + + roundup-admin help + +and see the help text. + +If you want to run Roundup commands in the future without +activating the virtual environment, just call the commands using the +full path. For example:: + + /path/to/environment/roundup/usr/bin/roundup-admin + +You can use the command ``deactivate`` to return to the normal +Python environment. However, for now continue with +`configuring your first tracker`_. + +Installing from downloaded source +--------------------------------- + +In general you should be installing from a released Roundup version +into a virtual environment. + +.. _current development version: ../code.html + +If you are installing a `current development version`_ or are a +developer or are an expert you can use the manual installation method +from a source install. From the unpacked source distribution, run:: + + sudo python3 setup.py install + +which will put the Roundup core code into your systems Python tree and +the command scripts into ``/usr/bin`` If you would like to place the Roundup scripts in a directory other than ``/usr/bin``, then specify the preferred location with ``--install-scripts``. For example, to install them in ``/opt/roundup/bin``:: - python setup.py install --install-scripts=/opt/roundup/bin - -You can also use the ``--prefix`` option to use a completely different -base directory, if you do not want to use administrator rights. If you -choose to do this, you may have to change Python's search path (sys.path) -yourself. - -Docker Support -~~~~~~~~~~~~~~ - -If you don't want to install it natively, you can create a Docker -container. This installs roundup using the `stand-alone web server`_ -method. This is an http only install so we suggest putting an https -terminating proxy in front of it. - -This is a work in progress and patches to improve it are welcome. You -can find the docker config files under the `scripts/Docker` directory -of the source tree. - -The dockerized Roundup includes database drivers for anydbm, sqlite, -MySQL and Postgresql (Postgresl is untested). It also includes -additional libraries that are listed in -`scripts/Docker/requirements.txt`. - -Email support is a work in progress. Outgoing email should work given -an external SMTP server. Receiving email should work by using a -scheduled (cron) job to access email: - -* `As a regular job using a mailbox source`_ -* `As a regular job using a POP source`_ -* `As a regular job using an IMAP source`_ - -Patches for better email support are welcome. - -If you want to use a MySQL backend, the `docker-compose.yml` file will -deploy a Roundup container and a MySQL container backend for use with -Roundup. - -Building a Docker Container -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To build a docker container using the code in the current directory, -run this build command from the top of the source tree:: - - docker build -t roundup-app -f scripts/Docker/Dockerfile . - -You can also build a container using the newest Roundup release on -PyPI, by running:: - - docker build -t roundup-app --build-arg="source=pypi" \ - -f scripts/Docker/Dockerfile . - -The docker declares a single volume mounted at -``/usr/src/app/tracker`` inside the container. You will mount your -tracker home directory at this location. The ``/usr/src/app`` path can -be changed by using ``--build-arg="appdir=/new/path"``. - -You can also add additional modules to the docker container by using -`--build-arg="pip_mod=requests setproctitle"`. - -Because of deficiencies in the docker program (see: -https://github.com/moby/moby/issues/29110#issuecomment-1100676306), -there is no way to determine the version of Python inside the -container and make that available as part of the build process. If -your build fails because the ``pythonversion does not match``, add the -suggested ``--build-arg`` to the ``docker build`` command line. - -By default the container runs Roundup using UID 1000. By setting -`--build-arg="roundup_uid=2000"` you can change the UID. - -Configuring Roundup in the Container -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Once the docker is created using one of the build commands above, run -an interactive session it with:: - - docker run -it --rm -p 9017:8080 \ - -v $PWD/tracker:/usr/src/app/tracker roundup-app:latest - -The ``-v`` option maps a directory from the host into the docker -container. Note that uid 1000 is used by roundup by default. The uid -of the directory (and all files under it) must match the uid. You can -set the UID at image build time, see above. This -example assumes your tracker configs are in the tracker -subdirectory. Replace ``$PWD/tracker`` with the full path name to the -directory where the tracker home(s) are to be stored. - -The ``-p`` option maps an external port (9017) to proxy the roundup -server running at port 8080 to the outside. - -If the tracker directory is empty, the docker container will prompt -you to install a tracker template and prompt you for the database -type. - -Then you need to configure the tracker by editing -``template/config.ini``. Make sure that the tracker web setting ends -in ``/issues/`` See `Configuring your first tracker` and the top of -``config.ini`` for other settings. - -Once you have configured the tracker, run another interactive session -with:: - - docker run --rm -it -p 9017:8080 \ - -v $PWD/tracker:/usr/src/app/tracker roundup-app:latest - -this will initialize the database and attempt to start the server. If -that is successful, use control-c to exit the server. - -Now start the server non-interactively (note no `-it` option) with:: - - docker run -p 9017:8080 \ - -v $PWD/tracker:/usr/src/app/tracker roundup-app:latest - -Your tracker will be available at: ``http://yourhost:9017/issues/``. - -If you need to access your container while the server is running you -can use:: - - docker exec -it c0d5 sh - -where ``c0d5`` is the id prefix for the running container obtained -from ``docker container ls``. - -If you add ``-e SHELL_DEBUG=1`` to the docker command, it sets the -``SHELL_DEBUG`` environment variable which will enable debugging -output from the startup script. - -Non-Guided Installation -''''''''''''''''''''''' - -If you got a tracker installed using the automatic setup above, you -can skip this section. To manually install and initialize the -trackers, you can get a shell without starting the roundup-server -using:: - - docker run -it \ - -v $PWD/tracker:/usr/src/app/tracker \ - --entrypoint sh roundup-app:latest - -Now you can configure your tracker using ``roundup-admin -i tracker`` -using the directions for `Configuring your first tracker`. - -Defining Multiple Trackers -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you want to run multiple trackers, create a subdirectory for each -tracker home under the volume mount point (``$PWD/tracker``). Then -invoke ``docker run`` passing the roundup-server tracker -specifications like:: - - docker run --rm -p 9017:8080 \ - -v /.../issue.tracker:/usr/src/app/tracker \ - roundup-app:latest tracker1=tracker/tracker1_home \ - tracker2=tracker/tracker2_home - -This will set up two trackers that can be reached at -``http://yourhost:9017/tracker1/`` and ``http://yourhost:9017/tracker2/``. -The arguments after roundup-app:latest are tracker paths that are -passed to roundup-server. - -Docker-compose Deployment -^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you want to run using the mysql backend, you can use docker-compose -with ``scripts/Docker/docker-compose.yml``. This will run Roundup and -MySQL in containers. Directions for building using docker-compose are -at the top of the yml file. + sudo python setup.py install --install-scripts=/opt/roundup/bin + +You can also use the ``--prefix`` option to install roundup into a +completely different base directory. If you choose to do this, you +will have to change Python's search path (sys.path) yourself. Configuring your first tracker ------------------------------- +============================== + +Make sure the ``roundup-admin`` script location is on your ``PATH`` +evironment variable. This is done automatically when you activate a +virtual environment. You can also specify the full path to the command +in the following steps. 1. To create a Roundup tracker (necessary to do before you can use the software in any real fashion), you need to set up a "tracker @@ -437,13 +411,9 @@ mkdir /opt/roundup/trackers - b. Either add the Roundup script location to your ``PATH`` - environment variable or specify the full path to - the command in the next step. - .. index:: roundup-admin; install subcommand - c. Install a new tracker with the command ``roundup-admin install``. + b. Install a new tracker with the command ``roundup-admin install``. You will be asked a series of questions. Descriptions of the provided templates can be found in `choosing your template`_ below. Descriptions of the available backends can be found in `choosing your backend`_ @@ -486,7 +456,7 @@ .. index:: roundup-admin; initialise subcommand - d. Initialise the tracker database with ``roundup-admin initialise``. + c. Initialise the tracker database with ``roundup-admin initialise``. You will need to supply an admin password at this step. You will be prompted:: @@ -505,7 +475,8 @@ 2. At this point, your tracker is set up, but doesn't have a nice user interface. To set that up, we need to `configure a web interface`_ and optionally `configure an email interface`_. If you want to try your - new tracker out, assuming "tracker :: web" is set to + new tracker out, assuming the ``web`` setting in the + ``[tracker]`` [3]_ section of config.ini is set to ``'http://localhost:8080/support/'``, run:: roundup-server support=/opt/roundup/trackers/support @@ -524,9 +495,19 @@ to run the server at port 1080 and bind to all ip addresses on your system. Then direct your web browser to ``http://your_host_name:1080/support/``. +.. [3] The rest of the documentation uses the abbreviated form "tracker :: + web" for specifying a section and setting. + Choosing Your Template ---------------------- +Roundup ships with 5 templates. A description of each follows. When +Roundup is installed, you can also get a description of available +templates using ``roundup-admin templates``. You can use this to view +additional templates that you create or download. + +.. _classic tracker: + Classic Template ~~~~~~~~~~~~~~~~ @@ -548,6 +529,12 @@ There are three other templates distributed with Roundup: +devel + This is a generic issue tracker that may be used to track bugs, + feature requests, project issues or any number of other types of + issues. Most users of Roundup will find that this template suits + them, with perhaps a few customisations. + responsive This issue tracker uses the same schema as devel. The difference between devel and responsive templates is the use of Twitter @@ -557,13 +544,6 @@ static files live (the subdirectory "static" in the default of the template). -devel - This is a generic issue tracker that may be used to track bugs, - feature requests, project issues or any number of other types of - issues. Most users of Roundup will find that this template suits - them, with perhaps a few customisations. - - jinja2 This is a generic issue tracker based on classic schema. It uses Jinja2 for templating and Twitter bootstrap for responsive markup. @@ -629,12 +609,12 @@ Configure a Web Interface -------------------------- +========================= There are multiple ways to deploy the web interface. If your tracker will be heavily used and accessible from the internet, we suggest using Apache or Nginx in WSGI mode or as a reverse proxy -to the stand alone web server or WSGI server like gunicorn. +to the stand alone web server or WSGI server like Gunicorn. A FastCGI deployment with an alternate web server is suitable for lower traffic sites. @@ -665,7 +645,7 @@ .. index:: pair: web interface; cgi Web Server cgi-bin -~~~~~~~~~~~~~~~~~~ +------------------ A benefit of using the cgi-bin approach is that it's the easiest way to restrict access to your tracker to only use HTTPS. Access will be slower @@ -674,6 +654,8 @@ If your Python isn't installed as "python" then you'll need to edit the ``roundup.cgi`` script to fix the first line. +.. index:: windows; IIS cgi installation + If you're using IIS on a Windows platform, you'll need to run this command for the cgi to work (it turns on the PATH_INFO cgi variable):: @@ -721,7 +703,7 @@ CGI-bin for Limited-Access Hosting -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +---------------------------------- If you are running in a shared-hosting environment or otherwise don't have permission to edit the system web server's configuration, but can create a @@ -764,12 +746,14 @@ .. index:: pair: web interface; stand alone server Stand-alone Web Server -~~~~~~~~~~~~~~~~~~~~~~ +---------------------- This approach will give you faster response than cgi-bin. You may investigate using ProxyPass or similar configuration in apache to have your tracker accessed through the same URL as other systems. +The stand alone serveris used by the Docker image. + The stand-alone web server is started with the command ``roundup-server``. It has several options - display them with ``roundup-server -h``. @@ -784,7 +768,7 @@ .. index:: pair: web interface; Zope Zope Product - ZRoundup -~~~~~~~~~~~~~~~~~~~~~~~ +----------------------- ZRoundup installs as a regular Zope product. Copy the ``share/roundup/frontends/ZRoundup`` (frontends/ZRoundup in the @@ -798,7 +782,7 @@ ! single: wsgi; apache Apache HTTP Server with mod_wsgi -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +-------------------------------- This is a work in progress thanks to Garth Jensen. @@ -809,7 +793,7 @@ detailed directions. Background -^^^^^^^^^^ +~~~~~~~~~~ These notes were developed on a Microsoft Azure VM running Ubuntu 18.04 LTS. The instructions below assume: @@ -823,7 +807,7 @@ - the server has a static public IP address of 11.11.11.101 Install mod-wsgi -^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~ You can install/build it using the python package manager pip, or install using the OS package manager (apt). @@ -889,7 +873,7 @@ to security issues in older releases. Configure web interface via wsgi_handler -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. In the tracker's home directory create a ``wsgi.py`` file with the following content (substituting ``/home/admin/trackers/mytracker`` @@ -902,7 +886,7 @@ application = RequestDispatcher(tracker_home) To run the tracker on Port 8000 as a foreground process -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +''''''''''''''''''''''''''''''''''''''''''''''''''''''' 1. Change the ``tracker.web`` url in ``config.ini`` to port 8000 at the server domain name or ip address (e.g. http://11.11.11.101:8000/). @@ -913,7 +897,7 @@ ``tracker.web`` to in ``config.ini``. Run tracker as background daemon -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +'''''''''''''''''''''''''''''''' To run the tracker on Port 80 or as a background process, you'll need to configure a UNIX group with appropriate privileges as described in @@ -933,7 +917,7 @@ ``sudo chmod -R g+sw ~/trackers/mytracker/db`` To run mod_wsgi on PORT 80 -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +'''''''''''''''''''''''''' 1. Change the ``tracker.web`` url in ``config.ini`` to the server url with no port designator. E.g. http://11.11.11.101. @@ -942,13 +926,13 @@ to port 80 run: ``sudo service apache2 stop``. To run as a foreground process -'''''''''''''''''''''''''''''' +++++++++++++++++++++++++++++++ 1. From the tracker home directory, run ``sudo mod_wsgi-express start-server wsgi.py --port 80 --user admin --group mytrackergrp`` To run as a background process -'''''''''''''''''''''''''''''' +++++++++++++++++++++++++++++++ 1. From the tracker home directory, bash ``sudo mod_wsgi-express setup-server wsgi.py --port=80 --user admin --group mytrackergrp --server-root=/etc/mod_wsgi-express-80`` @@ -991,33 +975,33 @@ Roundup apache interface uses the following options specified with ``PythonOption`` directives: - TrackerHome: - defines the tracker home directory - the directory that was specified - when you did ``roundup-admin init``. This option is required. - - TrackerLanguage: - defines web user interface language. mod_python applications do not - receive OS environment variables in the same way as command-line - programs, so the language cannot be selected by setting commonly - used variables like ``LANG`` or ``LC_ALL``. ``TrackerLanguage`` - value has the same syntax as values of these environment variables. - This option may be omitted. - - TrackerDebug: - run the tracker in debug mode. Setting this option to ``yes`` or - ``true`` has the same effect as running ``roundup-server -t debug``: - the database schema and used html templates are rebuilt for each - HTTP request. Values ``no`` or ``false`` mean that all html - templates for the tracker are compiled and the database schema is - checked once at startup. This is the default behaviour. - - TrackerTiming: - has nearly the same effect as environment variable ``CGI_SHOW_TIMING`` - for standalone roundup server. The difference is that setting this - option to ``no`` or ``false`` disables timings display. Value - ``comment`` writes request handling times in html comment, and - any other non-empty value makes timing report visible. By default, - timing display is disabled. +TrackerHome: + defines the tracker home directory - the directory that was specified + when you did ``roundup-admin init``. This option is required. + +TrackerLanguage: + defines web user interface language. mod_python applications do not + receive OS environment variables in the same way as command-line + programs, so the language cannot be selected by setting commonly + used variables like ``LANG`` or ``LC_ALL``. ``TrackerLanguage`` + value has the same syntax as values of these environment variables. + This option may be omitted. + +TrackerDebug: + run the tracker in debug mode. Setting this option to ``yes`` or + ``true`` has the same effect as running ``roundup-server -t debug``: + the database schema and used html templates are rebuilt for each + HTTP request. Values ``no`` or ``false`` mean that all html + templates for the tracker are compiled and the database schema is + checked once at startup. This is the default behaviour. + +TrackerTiming: + has nearly the same effect as environment variable ``CGI_SHOW_TIMING`` + for standalone roundup server. The difference is that setting this + option to ``no`` or ``false`` disables timings display. Value + ``comment`` writes request handling times in html comment, and + any other non-empty value makes timing report visible. By default, + timing display is disabled. In the following example we have two trackers set up in ``/var/db/roundup/support`` and ``/var/db/roundup/devel`` and accessed @@ -1071,6 +1055,8 @@ in which the tracker homes are stored. The actual value will thus depend on your system. +.. index:: windows; apache config + On Windows the corresponding lines will look similar to these:: AliasMatch /roundup/(.+)/@@file/(.*) C:/DATA/roundup/$1/html/$2 @@ -1091,13 +1077,13 @@ serving with SSL. Nginx HTTP Server -~~~~~~~~~~~~~~~~~ - -This configuration uses gunicorn to run roundup behind an Nginx proxy. +----------------- + +This configuration uses Gunicorn to run Roundup behind an Nginx proxy. The proxy also compresses the data using gzip. The url for the tracker in config.ini should be ``https://tracker.example.org``. - .. code:: +.. code:: user nginx; worker_processes auto; @@ -1111,7 +1097,7 @@ } upstream tracker-tracker { - # gunicorn uses this socket for communication + # Gunicorn uses this socket for communication server unix:/var/run/roundup/tracker.sock fail_timeout=0; } @@ -1205,7 +1191,7 @@ FastCGI (Cherokee, Hiawatha, lighttpd) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +-------------------------------------- The Hiawatha and lighttpd web servers can run Roundup using FastCGI. Cherokee can run FastCGI but it also supports wsgi directly using a @@ -1222,13 +1208,13 @@ .. _flup: https://pypi.org/project/flup/ WSGI Variations -~~~~~~~~~~~~~~~ +--------------- .. index:: triple: web interface; apache; mod_wsgi single: wsgi; apache Apache Alternate -^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~ This method from Thomas Arendsen Hein goes into a bit more detail and is designed to allow you to run multiple roundup trackers each under @@ -1250,7 +1236,7 @@ /etc/apache2/sites-available/roundup-foo (under debian/Ubunutu, modify as needed): - .. code:: ApacheConf +.. code:: ApacheConf ServerAdmin webmaster@example.com ErrorLog /var/log/apache2/error.log @@ -1289,13 +1275,13 @@ * a ``db`` subdirectory where messages and files will be stored * a symbolic link called ``instance`` to /srv/roundup/foo which has - been initialized using ``roundup-admin``. + been initialised using ``roundup-admin``. The `Apache HTTP Server with mod_wsgi`_ section above has a simple WSGI handler. This is an enhanced version to be put into ``/srv/roundup/foo/roundup.wsgi``. - .. code:: python +.. code:: python import sys, os sys.stdout = sys.stderr @@ -1329,14 +1315,14 @@ parameter in the [main] section to be /home/roundup-foo/db. This will put the files and messages in the db directory for the user. -.. index:: pair: web interface; gunicorn - single: wsgi; gunicorn +.. index:: pair: web interface; Gunicorn + single: wsgi; Gunicorn Gunicorn Installation -^^^^^^^^^^^^^^^^^^^^^ - -To run with gunicorn use pip to install gunicorn. This configuration -uses a front end web server like nginx, hiawatha, apache configured as +~~~~~~~~~~~~~~~~~~~~~ + +To run with Gunicorn use ``pip install gunicorn``. This configuration +uses a front end web server like nginx, hiawatha, or apache configured as a reverse proxy. See your web server's documentation on how to set it up as a reverse proxy. @@ -1354,7 +1340,7 @@ app = RequestDispatcher(tracker_home) -Assuming the proxy forwards /tracker, run gunicorn as:: +Assuming the proxy forwards /tracker, run Gunicorn as:: SCRIPT_NAME=/tracker gunicorn --bind 127.0.0.1:8917 --timeout=10 wsgi:app @@ -1368,7 +1354,7 @@ single: wsgi; uWSGI uWSGI Installation -^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~ For a basic roundup install using uWSGI behind a front end server, install uwsgi and the python3 (or python) plugin. Then run:: @@ -1377,12 +1363,12 @@ --plugin python3 --mount=/tracker=wsgi.py \ --manage-script-name --callable app -using the same wsgi.py as was used for gunicorn. If you get path not +using the same wsgi.py as was used for Gunicorn. If you get path not found errors, check the mount option. The /tracker entry must match the path used for the [tracker] web value in the tracker's config.ini. Configure an Email Interface ----------------------------- +============================ If you don't want to use the email component of Roundup, then remove the "``nosyreaction.py``" module from your tracker "``detectors``" directory. @@ -1394,7 +1380,7 @@ of which will continue my example setup from above: As a mail alias pipe process -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +---------------------------- Set up a mail alias called "issue_tracker" as (include the quote marks): "``|/usr/bin/python /usr/bin/roundup-mailgw <tracker_home>``" @@ -1411,13 +1397,15 @@ echo test |mail -s '[issue] test' support@YOUR_DOMAIN_HERE -Be careful that some mail systems (postfix for example) will impost a -limits on processes they spawn. In particular postfix can set a file size -limit. *This can cause your Roundup database to become corrupted.* +Be careful that some mail systems (postfix for example) will impose +limits on processes they spawn. In particular postfix can set a file +size limit that is inherited by the mailgw. If the database files +(anydbm, sqlite) exceed this limit, *this can cause your Roundup +database to become corrupted.* As a custom router/transport using a pipe process (Exim4 specific) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------------------------------------ The following configuration snippets for `Exim 4`_ configuration implement a custom router & transport to accomplish mail delivery to @@ -1451,7 +1439,7 @@ Exim4. .. note:: - Note that the Debian Exim4 packages don't allow pipes in alias files + The Debian Exim4 packages don't allow pipes in alias files by default, so the method described in the section `As a mail alias pipe process`_ will not work with the default configuration. However, the method described in this section does. See the discussion in @@ -1511,7 +1499,7 @@ group = ROUNDUP_GROUP As a regular job using a mailbox source -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------------------------------- Set ``roundup-mailgw`` up to run every 10 minutes or so. For example (substitute ``/usr/bin`` for wherever roundup-mailgw is installed):: @@ -1523,7 +1511,7 @@ will be "``/var/mail/issue_tracker``". As a regular job using a POP source -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +----------------------------------- To retrieve from a POP mailbox, use a *cron* entry similar to the mailbox one (substitute ``/usr/bin`` for wherever roundup-mailgw is @@ -1534,10 +1522,10 @@ where pop_spec is "``username:password@server``" that specifies the roundup submission user's POP account name, password and server. -On windows, you would set up the command using the windows scheduler. +On windows, you would set up the command `using the windows scheduler`_. As a regular job using an IMAP source -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------- To retrieve from an IMAP mailbox, use a *cron* entry similar to the POP one (substitute ``/usr/bin`` for wherever roundup-mailgw is @@ -1553,12 +1541,12 @@ If you have a secure (ie. HTTPS) IMAP server then you may use ``imaps`` in place of ``imap`` in the command to use a secure connection. -As with the POP job, on windows, you would set up the command using the -windows scheduler. +As with the POP job, on windows, you would set up the command +`using the windows scheduler`_. UNIX Environment Steps ----------------------- +====================== Each tracker ideally should have its own UNIX group, so create a UNIX group (edit ``/etc/group`` or your appropriate NIS map if @@ -1607,7 +1595,7 @@ Public Tracker Considerations ------------------------------ +============================= If you run a public tracker, you will eventually have to think about dealing with spam entered through both the web and mail interfaces. @@ -1616,12 +1604,320 @@ `customisation documentation`_ that has a simple detector that will block lot of spam attempts. +Docker Support +============== + +If you don't want to install Roundup on a host, you can create a +Docker container. This installs Roundup using the `stand-alone web +server`_ method. This image only supports http. We suggest putting an +https terminating proxy in front of it. + +This is a work in progress and patches to improve it are welcome. You +can find the docker config files under the `scripts/Docker` directory +of the source tree. + +The dockerized Roundup is based on a 64 bit Alpine distribution. It +includes database drivers for anydbm, sqlite, MySQL and Postgresql +(Postgresl is untested). It also includes additional libraries that +are listed in `scripts/Docker/requirements.txt` (including redis). + +Email support is a work in progress. Outgoing email to an external +SMTP server should work. Receiving email should work by using a +scheduled (cron) job to access email: + +* `As a regular job using a mailbox source`_ +* `As a regular job using a POP source`_ +* `As a regular job using an IMAP source`_ + +However running cron in a container is problematic (running +busybox crond as root vs. non-root, requiring setgrp privs +etc). Patches for implementing email support are welcome. + +If you want to use a MySQL backend, the `docker-compose.yml` file will +deploy a Roundup container and a MySQL container backend for use with +Roundup. + +We recommend you follow the `OSWAP Docker Security practices`_ for your +production Roundup instance. + +.. _OSWAP Docker Security practices: + https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html + +Building a Docker Container +--------------------------- + +To build a docker container using the code in the current directory, +run this build command from the top of the source tree:: + + docker build -t roundup-app -f scripts/Docker/Dockerfile . + +You can also build a container using the newest Roundup release on +PyPI, by running:: + + docker build -t roundup-app --build-arg="source=pypi" \ + -f scripts/Docker/Dockerfile . + +The docker declares a single volume mounted at +``/usr/src/app/tracker`` inside the container. You will mount your +tracker home directory at this location. The ``/usr/src/app`` path can +be changed by using ``--build-arg="appdir=/new/path"``. + +You can also add additional modules to the docker container by using +`--build-arg="pip_mod=requests setproctitle"`. + +Because of deficiencies in the docker program (see: +https://github.com/moby/moby/issues/29110#issuecomment-1100676306), +there is no way to determine the version of Python inside the +container and make that available as part of the build process. If +your build fails because the ``pythonversion does not match``, add the +suggested ``--build-arg`` to the ``docker build`` command line. + +.. _UID at image build time: + +By default the container runs Roundup using UID 1000. By setting +`--build-arg="roundup_uid=2000"` you can change the UID. + +Configuring Roundup in the Container +------------------------------------ + +.. caution:: + + Docker modifies iptables firewall rules. This allows access to the + container from your local network. `See the official documentation + for details + <https://docs.docker.com/engine/reference/commandline/run/#publish>`_. + UFW rules are known to be be ignored (see: + https://github.com/moby/moby/issues/4737). Use ``-p + 127.0.0.1:ext_port:container_port`` in your docker run commands or + implement suggestions like: https://github.com/chaifeng/ufw-docker. + +Once the docker image is created using one of the build commands +above, run an interactive session with:: + + docker run -it --rm -p 127.0.0.1:9017:8080 \ + -v $PWD/tracker:/usr/src/app/tracker roundup-app:latest + +The ``-v`` option maps a directory from the host into the docker +container. Note that uid 1000 is used by roundup by default. The uid +of the directory (and all files under it) must match the uid. You can +set the `UID at image build time`_. This +example assumes your tracker configs are in the tracker +subdirectory. Replace ``$PWD/tracker`` with the full path name to the +directory where the tracker home(s) are to be stored. + +The ``-p`` option maps an external port (9017) to proxy the roundup +server running at port 8080 to the outside. Note if you remove +``127.0.0.1:`` from the -p argument, *any host* on the network will be +able to access the tracker at port 9017. + +If the tracker directory is empty, the docker container will prompt +you to `install a tracker template`_ (step 3) and prompt you for the +database type. + +.. _install a tracker template: #configuring-your-first-tracker + +Then you need to configure the tracker by editing +``template/config.ini``. Make sure that the tracker web setting ends +in ``/issues/`` See `Configuring your first tracker`_ and the top of +``config.ini`` for other settings. + +Once you have configured the tracker, run another interactive session +to `initialise the tracker`_ (step 4) with:: + + docker run -it --rm -p 127.0.0.1:9017:8080 \ + -v $PWD/tracker:/usr/src/app/tracker roundup-app:latest + +this will initialise the database and attempt to start the server. If +that is successful, use control-c to exit the server. + +.. _initialise the tracker: #configuring-your-first-tracker + +Now start the server non-interactively (note no `-it` option) with:: + + docker run -p 9017:8080 -d \ + -v $PWD/tracker:/usr/src/app/tracker roundup-app:latest + +Your tracker will be available at: ``http://yourhost:9017/issues/``. + +If you need to access your container while the server is running you +can use:: + + docker exec -it c0d5 sh + +where ``c0d5`` is the id prefix for the running container obtained +from ``docker container ls``. + +You should place a web server in front of Roundup (in reverse proxy +mode) for production use. See the proxy_pass example below: + + * `Nginx HTTP Server`_ + +You can expose the port directly to your intranet by removing ``127.0.0.1`` +from the ``-p`` option. See `Stand-alone Web Server`_ for more details. + + +Running Your Container +---------------------- + +.. note:: + The examples below use the locally built docker container + specification: ``roundup-app``. You can replace it with the + docker hub specification ``rounduptracker/roundup:latest`` + (provided latest is newer than 2.3.0). + +As of version 2.3.0 the Docker container has multiple entry points. + +Guided install +~~~~~~~~~~~~~~ + +By default, running:: + + docker run -it --rm -p 127.0.0.1:9017:8080 \ + -v $PWD/tracker:/usr/src/app/tracker roundup-app:latest + +3 times will install, initialize and serve a Roundup tracker at +``..../issues/`` using ``$PWD/tracker`` as the tracker home. This is the +"guided install" method described in `Configuring Roundup in the +Container`_. + +Arguments for roundup-server +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once you have initialized your tracker, any arguments placed at the +end of the ``docker run`` command are passed to the ``roundup-server``. +These arguments **replace** the default arguments of ``issues=tracker``. + +Invoking a Shell +~~~~~~~~~~~~~~~~ + +You can invoke a shell inside the container without exec'ing into the +container using:: + + docker run -it \ + -v $PWD/tracker:/usr/src/app/tracker \ + roundup-app:latest shell + +Then you can manually configure your tracker using ``roundup-admin -i +tracker`` using the directions for `Configuring your first tracker`_. +This is also how you would access tools like ``roundup-gettext`` which +do not have direct entry points like ``admin`` for ``roundup-admin`` +and ``demo`` for ``roundup-demo``. + +Invoke roundup-admin +~~~~~~~~~~~~~~~~~~~~ + +You can run ``roundup-admin`` directly by using:: + + docker run -it \ + -v $PWD/tracker:/usr/src/app/tracker \ + roundup-app:latest admin -i tracker/tracker1 + +to start ``roundup-admin`` using the directory +``$PWD/tracker/tracker1``. This is one way to create multiple trackers +in subdirectories. It is no different from starting a shell and +invoking ``roundup-admin`` manually. + +One possibly useful command is:: + + + docker run -it \ + -v $PWD/tracker:/usr/src/app/tracker \ + roundup-app:latest admin templates + +to list description of all the installed templates. + +Invoke roundup-demo +~~~~~~~~~~~~~~~~~~~ + +Lastly you can:: + + docker run -it -p 127.0.0.1:8917:8080 \ + -v $PWD/tracker:/usr/src/app/tracker \ + roundup-app:latest demo anydbm responsive + +to create a directory ``$PWD/tracker/demo`` and autoconfigure a server +using the anydbm backend based on the responsive tracker template. +See `demo mode using docker`_ for steps to change the server port. + +Debugging +~~~~~~~~~ + +If you add ``-e SHELL_DEBUG=1`` to the docker command, it sets the +``SHELL_DEBUG`` environment variable which will enable debugging +output from the startup script. + +Running Multiple Trackers +-------------------------- + +If you want to run multiple trackers, create a subdirectory for each +tracker home under the volume mount point (``$PWD/tracker``). Then +invoke ``docker run`` passing the roundup-server tracker +specifications like:: + + docker run --rm -p 9017:8080 \ + -v /.../issue.tracker:/usr/src/app/tracker \ + roundup-app:latest tracker1=tracker/tracker1_home \ + tracker2=tracker/tracker2_home + +This will set up two trackers that can be reached at +``http://yourhost:9017/tracker1/`` and +``http://yourhost:9017/tracker2/``. The arguments after +``roundup-app:latest`` are arguments including tracker paths that are +passed to ``roundup-server``. + +Docker-compose Deployment +------------------------- + +If you want to run using the mysql backend, you can use docker-compose +with ``scripts/Docker/docker-compose.yml``. This will run Roundup and +MySQL in containers. Directions for building using docker-compose are +at the top of the yml file. + +Tags for Dockerhub Docker Images +-------------------------------- + +The docker images available from +https://hub.docker.com/r/rounduptracker/roundup +are tagged with: version-build, version, and ``latest`` tags. +For example, the tags when 2.3.0 is released will be: + +``rounduptracker/roundup:latest`` + is a moving tag that tracks the latest build + with the newest version of Roundup. + +``rounduptracker/roundup:2.3.0`` + is a moving tag that tracks the latest build + of version 2.3.0 of Roundup. The Roundup software in this + build will match the 2.3.0 version released on PyPi, but the + underlying Alpine image or versions of the supporting Python + libraries (redis, xapian, psycopg2, ...) will change. + +``rounduptracker/roundup:2.3.0-1`` + is a static tag and marks the first build of the version + 2.3.0 docker image. When a new release of the image is done, + it will get the tag ``2.3.0-2`` etc. This is an alternative to + pulling using a sha256 sum. However it is possible to + overwrite this image/tag. So it **does not** provide the same + security guarantees that using a sha256 sum does. + +In addition to the release tags, there may be one or more +development tags available. All tags will start with `devel`. For +example: ``rounduptracker/roundup:devel``. + +You should not assume that any ``devel`` tag is static. They ae +mainly for use by Roundup developer/maintainer for testing. There +may be alternate tags starting with ``devel-`` to indicate builds +from specific Mercurial versions/hashes. Also the tag may be +overwritten to change the underlying Python libraries or +images. Unless you like the bleeding edge, these should not be +used in production. Maintenance =========== -Read the separate `administration guide`_ for information about how to -perform common maintenance tasks with Roundup. +Read the `Tasks section of the administration guide +<admin_guide.html#tasks>`_ for information about how to perform +common maintenance tasks on Roundup. Upgrading @@ -1660,6 +1956,8 @@ Platform-Specific Notes ======================= +.. index:: windows; add Roundup to path + Windows command-line tools -------------------------- @@ -1691,6 +1989,9 @@ To have the Roundup web server start up when your machine boots up, there are two different methods, the scheduler and installing the service. +.. index:: windows; use scheduler for email integration + +.. _Using the Windows scheduler: 1. Using the Windows scheduler ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1698,35 +1999,31 @@ Set up the following in Scheduled Tasks (note, the following is for a cygwin setup): -**Run** - - ``c:\cygwin\bin\bash.exe -c "roundup-server TheProject=/opt/roundup/trackers/support"`` - -**Start In** - - ``C:\cygwin\opt\roundup\bin`` - -**Schedule** - - At System Startup +Run + ``c:\cygwin\bin\bash.exe -c "roundup-server TheProject=/opt/roundup/trackers/support"`` + +Start In + ``C:\cygwin\opt\roundup\bin`` + +Schedule + At System Startup To have the Roundup mail gateway run periodically to poll a POP email address, set up the following in Scheduled Tasks: -**Run** - - ``c:\cygwin\bin\bash.exe -c "roundup-mailgw /opt/roundup/trackers/support pop roundup:roundup@mail-server"`` - -**Start In** - - ``C:\cygwin\opt\roundup\bin`` - -**Schedule** - - Every 10 minutes from 5:00AM for 24 hours every day - - Stop the task if it runs for 8 minutes - +Run + ``c:\cygwin\bin\bash.exe -c "roundup-mailgw /opt/roundup/trackers/support pop roundup:roundup@mail-server"`` + +Start In + ``C:\cygwin\opt\roundup\bin`` + +Schedule + Every 10 minutes from 5:00AM for 24 hours every day + + Stop the task if it runs for 8 minutes + + +.. index:: windows; setup Roundup a service 2. Installing the roundup server as a Windows service ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1837,6 +2134,7 @@ .. _`customising roundup`: customizing.html .. _`upgrading document`: upgrading.html .. _`administration guide`: admin_guide.html +.. _`What does Roundup install`: admin_guide.html#what-does-roundup-install .. _`doc/postgresql.txt`: postgresql.html .. _`doc/mysql.txt`: mysql.html
--- a/roundup/demo.py Sun May 14 01:23:36 2023 -0400 +++ b/roundup/demo.py Sun May 14 09:43:53 2023 -0400 @@ -23,7 +23,7 @@ TRACKER_HOME = os.path.abspath('demo') -def install_demo(home, backend, template): +def install_demo(home, backend, template, use_port=None, use_host=None): """Install a demo tracker Parameters: @@ -91,9 +91,9 @@ os.remove(nosyreaction) # figure basic params for server - hostname = 'localhost' + hostname = use_host or 'localhost' # pick a fairly odd, random port - port = 8917 + port = use_port or 8917 while 1: print('Trying to set up web server on port %d ...' % port,) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) @@ -135,14 +135,32 @@ db.close() -def run_demo(home): - """Run the demo tracker instance from its ``home`` directory""" +def run_demo(home, bind_addr=None, bind_port=None): + """Run the demo tracker instance from its ``home`` directory + + For running under docker, we need to split ports into + the port roundup-server binds to (usually 8080) and the + external mapping requested by docker to be used in external url's. + """ + print("Demo Tracker Home:", home) cfg = configuration.CoreConfig(home) url = cfg["TRACKER_WEB"] - hostname, port = urlparse.urlparse(url)[1].split(':') - port = int(port) + try: + hostname, port = urlparse.urlparse(url)[1].split(':') + except ValueError: + print("\nThe TRACKER_WEB url:\n\n %(url)s\n\nin\n\n" + " %(home)s/config.ini" + "\n\nis missing a port number.\n" + "\nAre you using demo mode to start a production tracker? " + "Try running\ndemo mode with a different directory. " + "Use roundup-server to serve\nproduction trackers. " + "Exiting.\n" % {'home': home, 'url': url}) + exit(1) + + bind_addr = bind_addr or hostname + port = int(bind_port or port) success_message = '''Server running - connect to: %(url)s 1. Log in as "demo"/"demo" or "admin"/"admin". @@ -159,7 +177,7 @@ ''' % dict(url=url, script=sys.argv[0], datadir=TRACKER_HOME) # disable command line processing in roundup_server - sys.argv = sys.argv[:1] + ['-p', str(port), '-n', hostname, 'demo=' + home] + sys.argv = sys.argv[:1] + ['-p', str(port), '-n', bind_addr, 'demo=' + home] roundup_server.run(success_message=success_message)
--- a/roundup/scripts/roundup_demo.py Sun May 14 01:23:36 2023 -0400 +++ b/roundup/scripts/roundup_demo.py Sun May 14 09:43:53 2023 -0400 @@ -3,13 +3,9 @@ # Copyright 2004 Richard Jones (richard@mechanicalcat.net) # -DEFAULT_HOME = './demo' -DEFAULT_TEMPLATE = 'classic' - - +import argparse import sys - # --- patch sys.path to make sure 'import roundup' finds correct version import os.path as osp @@ -21,48 +17,171 @@ sys.path.insert(0, rootdir) # --/ +# import also verifies python version as side effect +from roundup import version_check # noqa: F401 E402 +from roundup import admin, configuration, demo, instance # noqa: E402 +from roundup import __version__ as roundup_version # noqa: E402 +from roundup.anypy.my_input import my_input # noqa: E402 +from roundup.backends import list_backends # noqa: E402 +from roundup.i18n import _ # noqa: E402 -from roundup import admin, configuration, demo, instance -from roundup.i18n import _ -from roundup.anypy.my_input import my_input -from roundup import version_check + +DEFAULT_HOME = './demo' +DEFAULT_TEMPLATE = 'classic' +DEFAULT_BACKEND = 'sqlite' +DEFAULT_PORT = 8917 + + +def usage(home, cli, msg=''): + + # massage the help. I want [directory [backend]] but there is no way to + # specify that in argparse, so replace the three positional args with + # the proper syntax. + usage = cli.format_help() % dict(locals()) + usage = usage.replace('[directory] [backend] [nuke]', + '[[directory] [backend]] [nuke]') + + print("%s\n" % usage) + + if msg: + print(msg) def run(): - home = DEFAULT_HOME - template = DEFAULT_TEMPLATE - nuke = sys.argv[-1] == 'nuke' + templates = admin.AdminTool().listTemplates().keys() + backends = list_backends() + + cli = argparse.ArgumentParser( + description= """ +Instant gratification demo - Roundup Issue Tracker + + Run a demo server. Config and database files are created in + 'directory' (current setting/default '%(home)s') which should + not exist or should exist and already be a tracker home + directory (usually used with nuke). + + 'nuke' will re-initialize the tracker instance, deleting the old data. + + The tracker that is created will have email notifications turned off. +""" % {"home": DEFAULT_HOME}, + epilog=("\nIf items marked with (*) are missing, they will be " + "asked for interactively when setting up the tracker."), + formatter_class=argparse.RawTextHelpFormatter, + add_help=True) + + cli.add_argument('-B', '--bind_address', + default="127.0.0.1", + help=( "Choose address for server to listen at.\n" + "Use 0.0.0.0 to bind to all addreses.\n" + "Default: %(default)s.\n\n")) + cli.add_argument('-b', '--backend_db', + choices=backends, + help=( "Choose backend database. Default: %s.\n\n" % + DEFAULT_BACKEND)) + cli.add_argument('-t', '--template', + choices=templates, + help="Use specified template. (*)\n\n") + cli.add_argument('-p', '--port', + type=int, + help=( "Listen at this port. Default: search for\n" + "open port starting at %s\n\n" % DEFAULT_PORT)) + cli.add_argument('-P', '--urlport', + type=int, + help=( "Set docker external port. If using\n" + " docker ... -p 9090:8917 ..." + "this should be set to 9090. " + "Default: as selected by --port\n\n")) + cli.add_argument('-V', '--version', action='version', + version='Roundup version %s'%roundup_version, + help=( + "Show program's version number: %s and exit\n" % + roundup_version)) + + cli.add_argument('directory', nargs='?', + help="Create home for tracker in directory. (*)\n") + + # add 'nuke' to choices so backend will accept nuke if only 2 args. + choices = backends + ['nuke'] + cli.add_argument('backend', nargs='?', metavar='backend', choices=choices, + help=( "Choose backend database. " + "Depricated, use -b instead.\n" + "If it is used, you *must* specify directory.\n\n")) + + cli.add_argument('nuke', nargs='?', metavar='nuke', choices=['nuke'], + help=( "The word 'nuke' will delete tracker and reset.\n" + "E.G. %(prog)s -b sqlite -t classic ./mytracker nuke\n") % {"prog": sys.argv[0]}) + + cli_args = cli.parse_args() + + # collect all positional args in order in array to parse + # strip all None. + cli_args.cmd = [ x for x in [cli_args.directory, cli_args.backend, cli_args.nuke] if x != None ] + + try: + nuke = cli_args.cmd[-1] == 'nuke' + if nuke: + _ignore = cli_args.cmd.pop() # remove nuke + except IndexError: + nuke = False + + try: + tracker_home = cli_args.cmd[0] + except IndexError: + tracker_home = None + + # invoked as demo tracker_dir sqlite [nuke] + try: + cli_backend = cli_args.cmd[1] + except IndexError: + cli_backend = None + + home = tracker_home or DEFAULT_HOME + template = cli_args.template or DEFAULT_TEMPLATE + backend = cli_args.backend_db or cli_backend or DEFAULT_BACKEND + # if there is no tracker in home, force nuke try: instance.open(home) + valid_home = True except configuration.NoConfigError: - nuke = 1 - # if we are to create the tracker, prompt for home + nuke = True + valid_home = False + + # if we are to create the tracker, prompt for settings if nuke: - if len(sys.argv) > 2: - backend = sys.argv[-2] - else: - backend = 'anydbm' # FIXME: i'd like to have an option to abort the tracker creation # say, by entering a single dot. but i cannot think of # appropriate prompt for that. - home = my_input( - _('Enter directory path to create demo tracker [%s]: ') % home) - if not home: - home = DEFAULT_HOME - templates = admin.AdminTool().listTemplates().keys() - template = my_input( - _('Enter tracker template to use (one of (%(template_list)s)) [%(d\ -efault_template)s]: ') % + if not tracker_home: + home = my_input( + _('Enter directory path to create demo tracker [%s]: ') % home) + if not home: + home = DEFAULT_HOME + + if not cli_args.template in templates: + template = my_input( + _('Enter tracker template to use (one of (%(template_list)s)) [%(default_template)s]: ') % { 'template_list': ','.join(templates), 'default_template': template}) - if not template: - template = DEFAULT_TEMPLATE + if not template: + template = DEFAULT_TEMPLATE + elif template not in templates: + print("Unknown template: %s. Exiting." % template) + exit(1) # install demo.install_demo(home, backend, - admin.AdminTool().listTemplates()[template]['path']) + admin.AdminTool().listTemplates()[template]['path'], + use_port=cli_args.urlport or DEFAULT_PORT) + else: + # make sure that no options are specified that are only useful on initialization. + if ( cli_args.backend or cli_args.template or + cli_args.backend_db ): + usage(home, cli, msg=( + "Specifying backend or template is only allowed when\n" + "creating a tracker or with nuke.\n")) + exit(1) # run - demo.run_demo(home) - + demo.run_demo(home, bind_addr=cli_args.bind_address, + bind_port=cli_args.port) if __name__ == '__main__': run()
--- a/scripts/Docker/roundup_healthcheck Sun May 14 01:23:36 2023 -0400 +++ b/scripts/Docker/roundup_healthcheck Sun May 14 09:43:53 2023 -0400 @@ -1,5 +1,7 @@ #! /bin/sh +# if there are multiple trackers, d=demo t=tracker ... +# returns last one for testing that server is up. Does not test +# each tracker. tracker=$(ps -ef | sed -ne '/roundup-server/s/^.*\s\(\w*\)=.*$/\1/p') -wget -q -O /dev/null --no-verbose http://localhost:8080/$tracker/ - +wget -q -O /dev/null --no-verbose http://localhost:8080/${tracker:-demo}/
--- a/scripts/Docker/roundup_start Sun May 14 01:23:36 2023 -0400 +++ b/scripts/Docker/roundup_start Sun May 14 09:43:53 2023 -0400 @@ -14,47 +14,163 @@ do_exit=0 +if test -t 0 -a -t 1; then # see if stdin/out are associated with a tty + might_be_interactive="true" +else + might_be_interactive="false" +fi + for tracker_spec in "$@"; do - # IFS== set a=b doesn't assign $1 and $2 in busybox ash - # it also clobbers '$@'. 'echo mumble | read' starts read in a + # IFS== set a=b doesn't just assign $1 and $2 in busybox ash + # it also clobbers '$@'. 'printf mumble | read' starts read in a # subshell so vars are not available in parent. IFS="=" read tracker directory <<- EOE - $tracker_spec + $tracker_spec EOE # ^ is a tab for use with <<- - # was $tracker_spec in the form of a=b, if not ignore it. - # allows setting CMD to -i index_template issue=tracker for example. - if [ -z "$directory" ]; then continue; fi + if [ -z "$directory" ]; then + # $tracker_spec was not in the form of a=b, check to see + # if it's a request to start in demo/shell mode: + case "$tracker" in + demo) + if [ -z "$PORT_8080" ]; then + PORT_8080=8917 + printf "If docker reports a bind error, you can set the\n" + printf "Docker environment variable PORT_8080 to fix it.\n" + printf "Please add -e PORT_8080=port_number. The\n" + printf "port_number must match the first value to -p \n" + printf "which must be an unused port on your server.\n" + fi + template=classic + backend=sqlite + + shift + + demoArgs=$# + + for arg in "$@"; do + # all keywords are unique, so iterate over all and + # assign as appropriate + case "$arg" in + classic|devel|jinja2|minimal|responsive) + template="$arg";; + + anydbm|sqlite) + backend="$arg";; + postgres|mysql) + printf "demo mode only supports sqlite or anydbm backends, not $1. Exiting." + exit 1;; + nuke) + nuke="$arg";; + *) printf "Unknown argument $1.\n" + printf "Usage: demo [template] [db]\n" + printf " template: one of " + printf "classic devel jinja2 minimal responsive\n" + printf " db: one of: sqlite anydbm\n" + printf "default: classic sqlite\n" + printf "Exiting\n" + exit 1 + esac + shift + done + + # run demo make sure to set bind address -B to 0.0.0.0 + # otherwise we can never make it out of the docker container. + # use -p to force port to match the exported docker port. + if [ -f tracker/demo/config.ini -a -z "$nuke" ]; then + if [ "$demoArgs" -ne 0 ]; then + printf "Error: backend and template arguments to demo " + printf "are invalid if a tracker\nis configured and " + printf "'nuke' is not specified.\nExiting.\n" + exit 1 + fi + printf "Restarting existing tracker.\n" + # Restart an existing demo tracker. + exec roundup-demo \ + -B 0.0.0.0 \ + -p 8080 \ + tracker/demo + else + # Create new or nuke/create existing tracker. + # Set parameters required to create tracker. + # Inherit the url port spec from the environment + # variable PORT_8080 using default value or value + # specified on the docker run command line. + printf "Creating new tracker.\n" + exec roundup-demo \ + -B 0.0.0.0 \ + -p 8080 \ + -b $backend \ + --urlport $PORT_8080 \ + -t $template \ + tracker/demo \ + $nuke + fi + ;; + shell) + if [ "$might_be_interactive" = "false" ]; then + printf \ + "Error: must use -it on docker command line to invoke shell\n" + exit 3 + fi + exec /bin/sh;; + admin) + shift + exec roundup-admin "$@";; + *) + # we just continue. Allow setting CMD to: + # -i index_template issue=tracker + # for example. + continue + esac + fi + + # we have a tracker=directory spec. Validate it and see if we need to + # install or initialize it. # something is specified or built wrong. Don't start. if [ ! -d "$directory" ]; then - printf "Unable to find directory %s. Exiting\n" "$directory" + printf "Unable to find directory %s for tracker %s. Exiting.\n" \ + "$directory" "$tracker" exit 1 fi - # user must define web in config.ini. - if ! grep '^\s*web\s\s*=\s\s*' "$directory/config.ini" > /dev/null; then + # verify that config.ini has been edited with a web spec. + # user must at minimum define web in config.ini. + if ! grep '^\s*web\s*=\s*' "$directory/config.ini" > /dev/null 2>&1 + then + if [ -e "$directory/config.ini" ]; then + printf "Please edit %s/config.ini and set the required +parameters.\n" "$directory" + printf "The web setting appears to be missing.\n" + exit 3 + fi + + printf "Installing %s tracker in %s\n" "$tracker" "$directory" roundup-admin -i "$directory" install do_exit=1 fi # we have a valid config.ini so init database if not done - if [ $do_exit == 0 -a ! -e "$directory/.init_done" ]; then - if roundup-admin -i "$directory" init; then - cat > "$directory/.init_done" <<- EOD - Don't delete this file. The docker startup needs it so it won't - re-initialze the database destroying all the data. - EOD - else - do_exit=1 + # if we get errors, the db directory should be missing + # and we print an error. + if [ $do_exit == 0 -a ! -e "$directory/db" ]; then + printf "Initializing tracker %s\n" "$tracker" + if ! roundup-admin -i "$directory" init; then + # something went wrong. + # verify it looks like a tracker directory + # then remove the database directory + test -e "$directory/TEMPLATE-INFO.txt" && \ + rm -rf "$directory/db" fi + do_exit=1 fi -done +done # for "$@" # if any config.ini needs editing don't start up. if [ $do_exit == 0 ]; then - # make roundup-server process 1 with exec + # make roundup-server process pid 1 with exec exec roundup-server -n 0.0.0.0 "$@" fi
--- a/share/man/man1/roundup-demo.1 Sun May 14 01:23:36 2023 -0400 +++ b/share/man/man1/roundup-demo.1 Sun May 14 09:43:53 2023 -0400 @@ -2,19 +2,82 @@ .SH NAME roundup-demo \- create a roundup "demo" tracker and launch its web interface .SH SYNOPSIS -\fBroundup-demo\fP [\fIbackend\fP [\fBnuke\fP]] +\fBroundup_demo\fP [\fB\-h\fP] [\fB-b\fP \fIBIND_ADDRESS\fP] +[\fB-b\fP {\fBanydbm\fP,\fBmysql\fP,\fBsqlite\fP,\fBpostgresql\fP}] +[\fB-t\fb +{\fBminimal\fP,\fBjinja2\fP,\fBclassic\fP,\fBresponsive\fP,\fBdevel\fP}] +[\fB-p\fP \fIPORT\fP] [\fB-P\fP \fIURLPORT\fP] [\fB-V\fP] +[\fIdirectory\fP] [\fIbackend\fP] [\fBnuke\fP] .SH OPTIONS .TP +\fIdirectory\fP + +The home directory for the new demo tracker. (*) +.TP +\fIbackend\fP + +Choose backend database. Depricated, use \fB-b\fP. +.TP \fBnuke\fP -Create a fresh demo tracker (deleting the existing one if any). If the -additional \fIbackend\fP argument is specified, the new demo tracker will -use the backend named (one of "anydbm", "sqlite", "mysql" or -"postgresql"; subject to availability on your system). + +Create a fresh demo tracker (deleting the existing one if +any). E.G. +.EX + roundup-demo -b sqlite -t classic ./mytracker nuke +.EE +will remove an existing tracker (if present) from the directory +\fB./mytracker\fP. Then it will create and serve a new empty classic +tracker using the sqlite backend. +.TP +\fB-h\fP, \fB--help\fP + +Show the help message and exit +.TP +\fB-b\fp \fIBIND_ADDRESS\fI, \fB--bind_address\fP \fIBIND_ADDRESS\fP + +Choose address for server to listen at. Use 0.0.0.0 to bind to all addreses. +Default: 127.0.0.1. +.TP +\fB-b\fP {\fBanydbm\fP,\fBmysql\fP,\fBsqlite\fP,\fBpostgresql\fP}, \ +\fB--backend_db\fP {\fBanydbm\fP,\fBmysql\fP,\fBsqlite\fP,\fBpostgresql\fP} + +Choose backend database. Default: +sqlite. Available backends are subject to availability on your system. +.TP +\fB-t\fP \ +{\fBminimal\fP,\fBjinja2\fP,\fBclassic\fP,\fBresponsive\fP,\fBdevel\fP}, \ +\fP--template\fB \ +{\fBminimal\fP,\fBjinja2\fP,\fBclassic\fP,\fBresponsive\fP,\fBdevel\fP} + +Use specified template when building tracker. (*) +.TP +\fB-p\fP \fIPORT\fP, \fB--port\fP \fIPORT\fP + +Listen at this port. Default: search for open port starting at 8917. +.TP +\fB-P\fP \fIURLPORT\fP, \fB--urlport\fP \fIURLPORT\fP When using + +docker this option passes the docker external port to the demo +instance. If using \fBdocker ... -p 9090:8917 ...\fP this should be +set to \fB-P 9090\fP. Default: as selected by \fB--port\fP. +.TP +\fB-V\fP, \fB--version\fP + +Show program's version number and exit + +.PP +If items marked with (*) are missing, they will be asked for +interactively when setting up the tracker. + .SH DESCRIPTION -This command creates a fresh demo tracker for you to experiment with. The -email features of Roundup will be turned off (so the nosy feature won't -send email). It does this by removing the \fInosyreaction.py\fP module -from the demo tracker's \fIdetectors\fP directory. +This command creates a fresh demo tracker for you to experiment +with. The email features of Roundup will be turned off (so the nosy +feature won't send email). It does this by removing the +\fBnosyreaction.py\fP module from the demo tracker's \fIdetectors\fP +directory. If you wish to send email to promote the trcker to +production, you will need to copy \fBnosyreaction.py\fP from the +default tracker templates directory. The \fBtemplates\fP command from +roundup-admin(1) can help you locate a replacement. If you wish, you may modify the demo tracker by editing its configuration files and HTML templates. See the \fIcustomisation\fP manual for @@ -25,6 +88,8 @@ command to install the tracker from inside the demo tracker home directory, and it will be listed as an available template for installation. No data will be copied over. +.SH SEE ALSO + .SH AUTHOR -This manpage was written by Richard Jones -<richard@users.sourceforge.net>. +This manpage was written by Richard Jones and extensively modified by +John Rouillard <rouilj@users.sourceforge.net>.