Roundup (0.3.1)

An Issue-Tracking System for Knowledge Workers

Contents

• Overview (Initial submission to SC Track)
• Installation
  • Prerequisites
  • Getting Roundup
  • Installing Roundup
• Getting Started
  • The Instance
  • Command Line Tool
  • E-Mail Interface
  • Web Interface
  • Users and Access Control (Users and permissions, Adding users)
  • Issues
• User Guide
  • Command Line Tool
  • Web Interface
  • E-Mail Gateway (Message content summary, Address handling, Performing Actions)
• Customising Roundup
  • Instance Configuration
  • Instance Schema (Creating a new information store)
  • Web Interface (Displaying properties, Index views, Item views)
• Roundup's Design Document ("Implementation Guide")
• Acknowledgements

Installation

Prerequisites

Python 2.1.1 is required for the correct operation of roundup.

Download the latest version from http://www.python.org/.

Getting Roundup

Installing Roundup

1. Run:
   python setup.py install
2. If you would prefer the scripts installed in somewhere other than /usr/local/bin, add "--install-scripts=<dir>" to the command:
   python setup.py install --install-scripts=<dir>
3. The command:
   python setup.py install --help
   gives all the options available for installation.

Getting Started

The Instance

1. Hyperdatabase
   This is the lowest component of Roundup and is where all the issues, users, file attachments and messages are stored.
2. Database schema
   This describes the content of the hyperdatabase - what fields are stored for issues, what user information, etc. Being stored in the instance, this allows it to be customised for a particular application. It also means that changes in the Roundup core code do not affect a running instance.
3. Web Interface
   The web interface templates are defined in the instance too - and the actual CGI interface class is defined (mostly using base classes in the Roundup core code) so it, like the database, may be customised for each instance in use.

Command Line Tool

1. Instance home directory
2. Schema to use
3. Database back-end to use
4. Administration user "admin" password.

Roundup is configurable using an instance_config.py file in the instance home. It should be edited before roundup is used, and may have the following variable declarations:

1. MAILHOST
   The SMTP mail host that roundup will use to send mail
2. MAIL_DOMAIN
   The domain name used for email addresses
3. ISSUE_TRACKER_WEB
   The web address of the issue tracker's web interface

The email addresses used by the system by default are:

1. ISSUE_TRACKER_EMAIL - issue_tracker@MAIL_DOMAIN
   submissions of issues
2. ADMIN_EMAIL - roundup-admin@MAIL_DOMAIN
   roundup's internal use (problems, etc)

E-Mail Interface

Setup 1: As a mail alias pipe process

"|/usr/bin/python /usr/local/bin/roundup-mailgw <instance_home>"

|roundup-mailgw <instance_home>

echo test |mail -s '[issue] test' issue_tracker@your.domain

Setup 2: As a regular cron job using a mailbox source

10 * * * * /usr/local/bin/roundup-mailgw <instance_home> mailbox <mail_spool_file>

Setup 3: As a regular cron job using a POP source

10 * * * * /usr/local/bin/roundup-mailgw <instance_home> pop <pop_spec>

Web Interface

Stand-alone:

1. Edit roundup-server at the top - ROUNDUP_INSTANCE_HOMES needs to know about your instance. You may also specify the values for ROUNDUP_INSTANCE_HOMES on the command-line using "name=home" pairs.
2. roundup-server [hostname port] (hostname may be "")
3. Load up the page /<instance name>/index where instance name is the name you nominated in ROUNDUP_INSTANCE_HOMES.

1. The CGI script is found in the cgi-bin directory of the roundup distribution.
2. Make sure roundup.cgi is executable. Edit it at the top - ROUNDUP_INSTANCE_HOMES needs to know about your instance.
3. Edit your /etc/httpd/conf/httpd.conf and make sure that the /home/httpd/html/roundup/roundup.cgi script will be treated as a CGI script.
4. Re-start your apache to re-load the config if necessary.
5. Load up the page "/roundup/roundup.cgi//index" where instance name is the name you nominated in ROUNDUP_INSTANCE_HOMES.
6. To use the CGI script unchanged, which allows much easier updates, add these directives to your "httpd.conf":

           SetEnv ROUNDUP_LOG "/var/log/roundup.log"
           SetEnv ROUNDUP_INSTANCE_HOMES "Default=/usr/local/share/roundup/instances/Default"
           SetEnv ROUNDUP_DEBUG "0"
   

7. On Windows, write a batch file "roundup.bat" similar to the one below and place it into your cgi-bin directory:

           @echo off
           set ROUNDUP_LOG=c:\Python21\share\roundup\cgi.log
           set ROUNDUP_INSTANCE_HOMES=Default=c:\Python21\share\roundup\instances\Default;
           set ROUNDUP_DEBUG=0
           c:\Python21\python.exe c:\Python21\share\roundup\cgi-bin\roundup.cgi
   

Users

Users and permissions

If users attempt to use roundup in any manner and are not identified to roundup, they will be using the database in a read-only mode. That is, if roundup doesn't know who they are, they can't change anything. This has the following repurcussions:

Command-line interface

The data modification commands (create, init, retire, set) are performed as the "admin" user. It is therefore important that the database be protected by the filesystem if protection is required. On a Unix system, the easiest and most flexible method of doing so is:

1. Add a new user and group to your system (e.g. "issue_tracker")
2. When creating a new instance home, use the following commands (substituting instance_home for the directory you want to use):
   

   mkdir instance_home
   chown issue_tracker:issue_tracker instance_home
   chmod g+rwxs instance_home
   roundup-admin -i instance_home init
   

3. Now, edit the /etc/group line for the issue_tracker group so it includes the unix logins of all the users who are going to administer your roundup instance. If you're running the web or mail gateways, then be sure to include those users in the group too (on some Linux systems, these users are "www" or "apache" and "mail".)

E-Mail interface

Users are identified by e-mail address - a new user entry will be created for any e-mail address that is not recognised, so users are always identified by roundup.

Web interface

Unidentified users have read-only access. If the users database has an entry with the username "anonymous", then unidentified users are automatically logged in as that user. This gives them write access.

*** anonymous access and the ANONYMOUS_* configuratins.

Adding users

1. On the web, access the URL .../<instance name>/newuser to bring up a form which may be used to add a new user.
2. On the command-line, use:
   roundup-admin -i <instance home> create user username=bozo password=bozo address=richard@clown.org
   Supply the admin username and password. roundup-admin will print the id of the new user.
3. Any e-mail sent to roundup from an address that doesn't match an existing user in the database will result in a new user entry being created for that user.

Issues

1. On the web, access the URL .../<instance name>/newissue to bring up a form which may be used to add a new issue.
2. On the command-line, use:
   roundup-admin -i <instance home> create issue title="test issue"
   Supply the admin username and password. roundup-admin will print the id of the new issue.
3. Any e-mail sent to roundup with the subject line containing [issue] will automatically created a new issue in the database using the contents of the e-mail.

User Guide

Command Line Tool

Options:
-i instance home	specify the issue tracker "home directory" to administer
-u	the user[:password] to use for commands
-c	when outputting lists of data, just comma-separate them

Command Help
commit	Usage: commit The changes made during an interactive session are not automatically written to the database - they must be committed using this command. One-off commands on the command-line are automatically committed if they are successful.
create	Usage: create classname property=value ... This creates a new entry of the given class using the property name=value arguments provided on the command line after the "create" command.
display	Usage: display designator This lists the properties and their associated values for the given node.
export	Usage: export class[,class] destination_dir This action exports the current data from the database into tab-separated-value files that are placed in the nominated destination directory. The journals are not exported.
find	Usage: find classname propname=value ... Find the nodes of the given class with a given link property value. The value may be either the nodeid of the linked node, or its key value.
get	Usage: get property designator[,designator]* Retrieves the property value of the nodes specified by the designators.
help	Usage: help topic commands -- list commands -- help specific to a command initopts -- init command options all -- all available help
history	Usage: history designator Lists the journal entries for the node identified by the designator.
import	Usage: import class file The file must define the same properties as the class (including having a "header" line with those property names.) The new nodes are added to the existing database - if you want to create a new database using the imported data, then create a new database (or, tediously, retire all the old data.)
initialise	Usage: initialise [template [backend [admin password]]] The command will prompt for the instance home directory (if not supplied through INSTANCE_HOME or the -i option. The template, backend and admin password may be specified on the command-line as arguments, in that order. See also initopts help.
list	Usage: list classname [property] Lists all instances of the given class. If the property is not specified, the "label" property is used. The label property is tried in order: the key, "name", "title" and then the first property, alphabetically.
retire	Usage: retire designator[,designator]* This action indicates that a particular node is not to be retrieved by the list or find commands, and its key value may be re-used.
rollback	Usage: rollback The changes made during an interactive session are not automatically written to the database - they must be committed manually. This command undoes all those changes, so a commit immediately after would make no changes to the database.
set	Usage: set designator[,designator]* propname=value ... Sets the property to the value for all designators given.
specification	Usage: specification classname This lists the properties for a given class.
table	Usage: table classname [property[,property]*] Lists all instances of the given class. If the properties are not specified, all properties are displayed. By default, the column widths are the width of the property names. The width may be explicitly defined by defining the property as "name:width". For example:: roundup> table priority id,name:10 Id Name 1 fatal-bug 2 bug 3 usability 4 feature

All commands (except help) require an instance specifier. This is just the path to the roundup instance you're working with. A roundup instance is where roundup keeps the database and configuration file that defines an issue tracker. It may be thought of as the issue tracker's "home directory". It may be specified in the environment variable ROUNDUP_INSTANCE or on the command line as "-i instance".

A designator is a classname and a nodeid concatenated, eg. bug1, user10, ...

Property values are represented as strings in command arguments and in the printed results:

• Strings are, well, strings.
• Password values will display as their encoded value.
• Date values are printed in the full date format in the local time zone, and accepted in the full format or any of the partial formats explained below.

  Input of...	Means...
  "2000-04-17.03:45"	2000-04-17.08:45:00
  "2000-04-17"	2000-04-17.00:00:00
  "01-25"	yyyy-01-25.00:00:00
  "08-13.22:13"	yyyy-08-14.03:13:00
  "11-07.09:32:43"	yyyy-11-07.14:32:43
  "14:25"	yyyy-mm-dd.19:25:00
  "8:47:11"	yyyy-mm-dd.13:47:11
  "."	"right now"

• Link values are printed as node designators. When given as an argument, node designators and key strings are both accepted.
• Multilink values are printed as lists of node designators joined by commas. When given as an argument, node designators and key strings are both accepted; an empty string, a single node, or a list of nodes joined by commas is accepted.

When multiple results are returned by the roundup get or roundup find commands, they are printed one per line (default) or joined by commas (with the -c) option.

Where the command changes data, a login name/password is required. The login may be specified as either "name" or "name:password".

• ROUNDUP_LOGIN environment variable
• the -u command-line option

Web Interface

    :sort    - sort by prop name, optionally preceeded with '-'
             to give descending or nothing for ascending sorting.
    :group   - group by prop name, optionally preceeded with '-' or
             to sort in descending or nothing for ascending order.
    :filter  - selects which props should be displayed in the filter
             section. Default is all.
    :columns - selects the columns that should be displayed.
             Default is all.
    propname - selects the values the node properties given by propname
             must have (very basic search/filter).

E-Mail Gateway

Performing Actions

If an item designator (class name and id number) is found there, the newly created "msg" node is added to the "messages" property for that item, and any new "file" nodes are added to the "files" property for the item.

If just an item class name is found there, we attempt to create a new item of that class with its "messages" property initialized to contain the new "msg" node and its "files" property initialized to contain any new "file" nodes.

Setting Properties

Message content

• In a multipart/mixed message or part, each subpart is extracted and examined. The text/plain subparts are assembled to form the textual body of the message, to be stored in the file associated with a "msg" class node. Any parts of other types are each stored in separate files and given "file" class nodes that are linked to the "msg" node.
• In a multipart/alternative message or part, we look for a text/plain subpart and ignore the other parts.

Message summary

Address handling

Triggers

Nosy Lists

Customising Roundup

Instance Configuration

MAIL_DOMAIN=MAILHOST=HTTP_HOST=None
HTTP_PORT=0

# roundup home is this package's directory
INSTANCE_HOME=os.path.split(__file__)[0]

# The SMTP mail host that roundup will use to send mail
if not MAILHOST:
    MAILHOST = 'localhost'

# The domain name used for email addresses.
if not MAIL_DOMAIN:
    MAIL_DOMAIN = 'fill.me.in.'

# the next two are only used for the standalone HTTP server.
if not HTTP_HOST:
    HTTP_HOST = ''
if not HTTP_PORT:
    HTTP_PORT = 9080

# This is the directory that the database is going to be stored in
DATABASE = os.path.join(INSTANCE_HOME, 'db')

# This is the directory that the HTML templates reside in
TEMPLATES = os.path.join(INSTANCE_HOME, 'html')

# The email address that mail to roundup should go to
ISSUE_TRACKER_EMAIL = 'issue_tracker@%s'%MAIL_DOMAIN

# The web address that the instance is viewable at
ISSUE_TRACKER_WEB = 'http://some.useful.url/'

# The email address that roundup will complain to if it runs into trouble
ADMIN_EMAIL = 'roundup-admin@%s'%MAIL_DOMAIN

# Somewhere for roundup to log stuff internally sent to stdout or stderr
LOG = os.path.join(INSTANCE_HOME, 'roundup.log')

# Where to place the web filtering HTML on the index page
FILTER_POSITION = 'bottom'      # one of 'top', 'bottom', 'top and bottom'

# Deny or allow anonymous access to the web interface
ANONYMOUS_ACCESS = 'deny'

# Deny or allow anonymous users to register through the web interface
ANONYMOUS_REGISTER = 'deny'

# Send nosy messages to the author of the message
MESSAGES_TO_AUTHOR = 'no'       # either 'yes' or 'no'

Instance Schema

    pri = Class(db, "priority", name=String(), order=String())
    pri.setkey("name")
    pri.create(name="critical", order="1")
    pri.create(name="urgent", order="2")
    pri.create(name="bug", order="3")
    pri.create(name="feature", order="4")
    pri.create(name="wish", order="5")

    stat = Class(db, "status", name=String(), order=String())
    stat.setkey("name")
    stat.create(name="unread", order="1")
    stat.create(name="deferred", order="2")
    stat.create(name="chatting", order="3")
    stat.create(name="need-eg", order="4")
    stat.create(name="in-progress", order="5")
    stat.create(name="testing", order="6")
    stat.create(name="done-cbb", order="7")
    stat.create(name="resolved", order="8")

    keyword = Class(db, "keyword", name=String())
    keyword.setkey("name")

    user = Class(db, "user", username=String(), password=String(),
        address=String(), realname=String(), phone=String(), organisation=String())
    user.setkey("username")
    user.create(username="admin", password=adminpw, address=instance_config.ADMIN_EMAIL)

    msg = FileClass(db, "msg", author=Link("user"), recipients=Multilink("user"), 
        date=Date(), summary=String(), files=Multilink("file"))

    file = FileClass(db, "file", name=String(), type=String())

    issue = IssueClass(db, "issue", assignedto=Link("user"),
        topic=Multilink("keyword"), priority=Link("priority"), status=Link("status"))
    issue.setkey('title')

Classes and Properties - creating a new information store

priority

Defines the possible levels of urgency for issues.

status

Defines the possible states of processing the issue may be in.

keyword

Initially empty, will hold keywords useful for searching issues.

user

Initially holding the "admin" user, will eventually have an entry for all users using roundup.

msg

Initially empty, will all e-mail messages sent to or generated by roundup.

file

Initially empty, will all files attached to issues.

issue

Initially emtyp, this is where the issue information is stored.

We define the "priority" and "status" classes to allow two things: reduction in the amount of information stored on the issue and more powerful, accurate searching of issues by priority and status. By only requiring a link on the issue (which is stored as a single number) we reduce the chance that someone mis-types a priority or status - or simply makes a new one up.

Class and Nodes

The actual data entered into the database, using class.create() are called nodes. They have a special immutable property called id. We sometimes refer to this as the nodeid.

Properties

• String properties are for storing arbitrary-length strings.
• Password properties are for storing encoded arbitrary-length strings. The default encoding is defined on the roundup.password.Password class.
• Date properties store date-and-time stamps. Their values are Timestamp objects.
• A Link property refers to a single other node selected from a specified class. The class is part of the property; the value is an integer, the id of the chosen node.
• A Multilink property refers to possibly many nodes in a specified class. The value is a list of integers.

FileClass

IssueClass

The messages and files properties list the links to the messages and files related to the issue. The nosy property is a list of links to users who wish to be informed of changes to the issue - they get "CC'ed" e-mails when messages are sent to or generated by the issue. The nosy reactor (in the detectors directory) handles this action. The superceder link indicates an issue which has superceded this one.

They also have the dynamically generated "creation", "activity" and "creator" properties.

The value of the "creation" property is the date when a node was created, and the value of the "activity" property is the date when any property on the node was last edited (equivalently, these are the dates on the first and last records in the node's journal). The "creator" property holds a link to the user that created the issue.

setkey(property)

roundup-admin set issue assignedto=2

or

roundup-admin set issue assignedto=richard

Note, the same thing can be done in the web and e-mail interfaces.

create(information)

Web Interface

Most customisation of the web view can be done by modifying the templates in the instance html directory. These are divided into index, item and newitem views. The newitem view is optional - the item view will be used if the newitem view doesn't exist.

Displaying Properties

Properties appear in the user interface in three contexts: in indices, in editors, and as filters. For each type of property, there are several display possibilities. For example, in an index view, a string property may just be printed as a plain string, but in an editor view, that property should be displayed in an editable field.

The display of a property is handled by functions in the htmltemplate module.

Displayer functions are triggered by <display> tags in templates. The call attribute of the tag provides a Python expression for calling the displayer function. The three standard arguments are inserted in front of the arguments given. For example, the occurrence of

    <display call="plain('status')">

The displayer functions are
plain	Display a String property directly. Display a Date property in a specified time zone with an option to omit the time from the date stamp. For a Link or Multilink property, display the key strings of the linked nodes (or the ids if the linked class has no key property). Options: escape (boolean) - HTML-escape the resulting text.
field	Display a property like the plain displayer above, but in a form field to be edited. Strings, Dates and Intervals use TEXT fields, Links use SELECT fields and Multilinks use SELECT MULTIPLE fields. Options: size (number) - width of TEXT fields. height (number) - number of nows in SELECT MULTIPLE tags. showid (boolean) - true includes the id of linked nodes in the SELECT MULTIPLE fields.
menu	For a Links and Multilinks, display the same field as would be generated using field.
link	For a Link or Multilink property, display the names of the linked nodes, hyperlinked to the item views on those nodes. For other properties, link to this node with the property as the text. Options: property (property name) - the property to use in the second case. is_download (boolean) - to be used for links te file class nodes - this indicates that the URL should have the specified property (usually name) appended so that the download has the correct name (instead of the node designator.)
count	For a Multilink property, display a count of the number of links in the list. Arguments: property (property name) - the property to use.
reldate	Display a Date property in terms of an interval relative to the current date (e.g. "+ 3w", "- 2d"). Arguments: property (property name) - the property to use. Options: pretty (boolean) - display the relative date in an English form.
download	Show a Link("file") or Multilink("file") property using links that allow you to download files. Arguments: property (property name) - the property to use.
checklist	For a Link or Multilink property, display checkboxes for the available choices to permit filtering. Arguments: property (property name) - the property to use.
note	Display the special notes field, which is a text area for entering a note to go along with a change.
list	List the nodes specified by property using the standard index for the class. Arguments: property (property name) - the property to use.
history	List the history of the item.
submit	Add a submit button for the item.

Index Views

An index view contains two sections: a filter section and an index section. The filter section provides some widgets for selecting which items appear in the index. The index section is a table of items.

Index View Specifiers

An index view specifier (URL fragment) looks like this (whitespace has been added for clarity):

/issue?status=unread,in-progress,resolved&
        topic=security,ui&
        :group=+priority&
        :sort=-activity&
        :filters=status,topic&
        :columns=title,status,fixer

The index view is determined by two parts of the specifier: the layout part and the filter part. The layout part consists of the query parameters that begin with colons, and it determines the way that the properties of selected nodes are displayed. The filter part consists of all the other query parameters, and it determines the criteria by which nodes are selected for display.

The filter part is interactively manipulated with the form widgets displayed in the filter section. The layout part is interactively manipulated by clicking on the column headings in the table.

The filter part selects the union of the sets of items with values matching any specified Link properties and the intersection of the sets of items with values matching any specified Multilink properties.

The example specifies an index of "issue" nodes. Only items with a "status" of either "unread" or "in-progres" or "resolved" are displayed, and only items with "topic" values including both "security" and "ui" are displayed. The items are grouped by priority, arranged in ascending order; and within groups, sorted by activity, arranged in descending order. The filter section shows filters for the "status" and "topic" properties, and the table includes columns for the "title", "status", and "fixer" properties.

Associated with each item class is a default layout specifier. The layout specifier in the above example is the default layout to be provided with the default bug-tracker schema described above in section 4.4.

Filter Section

The template for a filter section provides the filtering widgets at the top of the index view. Fragments enclosed in <property>...</property> tags are included or omitted depending on whether the view specifier requests a filter for a particular property.

A property must appear in the filter template for it to be available as a filter.

Here's a simple example of a filter template.

<property name=status>
    <display call="checklist('status')">
</property>
<br>
<property name=priority>
    <display call="checklist('priority')">
</property>
<br>
<property name=fixer>
    <display call="menu('fixer')">
</property>

The standard index generation code appends a section to the index pages which allows selection of the filters - from those which are defined in the filter template.

Index Section

The template for an index section describes one row of the index table. Fragments enclosed in <property>...</property> tags are included or omitted depending on whether the view specifier requests a column for a particular property. The table cells should contain <display> tags to display the values of the item's properties.

Here's a simple example of an index template.

<tr>
    <property name=title>
        <td><display call="plain('title', max=50)"></td>
    </property>
    <property name=status>
        <td><display call="plain('status')"></td>
    </property>
    <property name=fixer>
        <td><display call="plain('fixer')"></td>
    </property>
</tr>

Sorting

String and Date values are sorted in the natural way. Link properties are sorted according to the value of the "order" property on the linked nodes if it is present; or otherwise on the key string of the linked nodes; or finally on the node ids. Multilink properties are sorted according to how many links are present.

Item Views

An item view contains an editor section and a spool section. At the top of an item view, links to superseding and superseded items are always displayed.

Editor Section

The editor section is generated from a template containing <display> tags to insert the appropriate widgets for editing properties.

Here's an example of a basic editor template.

<table>
<tr>
    <td colspan=2>
        <display call="field('title', size=60)">
    </td>
</tr>
<tr>
    <td>
        <display call="field('fixer', size=30)">
    </td>
    <td>
        <display call="menu('status')>
    </td>
</tr>
<tr>
    <td>
        <display call="field('nosy', size=30)">
    </td>
    <td>
        <display call="menu('priority')>
    </td>
</tr>
<tr>
    <td colspan=2>
        <display call="note()">
    </td>
</tr>
</table>

As shown in the example, the editor template can also request the display of a "note" field, which is a text area for entering a note to go along with a change.

When a change is submitted, the system automatically generates a message describing the changed properties.

If a note is given in the "note" field, the note is appended to the description. The message is then added to the item's message spool (thus triggering the standard detector to react by sending out this message to the nosy list).

The message also displays all of the property values on the item and indicates which ones have changed. An example of such a message might be this:

Polly's taken a turn for the worse - this is now really important!
-----
title: Polly Parrot is dead
priority: critical
status: unread -> in-progress
fixer: terry
keywords: parrot,plumage,perch,nailed,dead

Spool Section

The spool section lists messages in the item's "messages" property. The index of messages displays the "date", "author", and "summary" properties on the message nodes, and selecting a message takes you to its content.

The <property> tag used in the index may also be used here - it checks to see if the nominated Multilink property has any entries. This can be used to eliminate sections of the spool section if the property has no entries.

<property name="files">
 <tr class="strong-header">
  <td><b>Files</b></td>
 </tr>

 <tr>            
  <td><display call="list('files')"></td>
 </tr>
</property>

Acknowledgements