Mercurial Repository: p/roundup/code: doc/design.txt comparison

comparison doc/design.txt @ 1356:83f33642d220 maint-0.5

[[Metadata associated with this commit was garbled during conversion from CVS to Subversion.]]

author	Richard Jones <richard@users.sourceforge.net>
date	Thu, 09 Jan 2003 22:59:22 +0000
parents
children

comparison

equal deleted inserted replaced

-:3d0158c8c32b
+:83f33642d220
+========================================================
+Roundup - An Issue-Tracking System for Knowledge Workers
+========================================================
+:Authors: Ka-Ping Yee (original__), Richard Jones (implementation)
+__ spec.html
+.. contents::
+Introduction
+---------------
+This document presents a description of the components
+of the Roundup system and specifies their interfaces and
+behaviour in sufficient detail to guide an implementation.
+For the philosophy and rationale behind the Roundup design,
+see the first-round Software Carpentry submission for Roundup.
+This document fleshes out that design as well as specifying
+interfaces so that the components can be developed separately.
+The Layer Cake
+-----------------
+Lots of software design documents come with a picture of
+a cake.  Everybody seems to like them.  I also like cakes
+(i think they are tasty).  So i, too, shall include
+a picture of a cake here::
+_________________________________________________________________________
+|  E-mail Client   |   Web Browser   |   Detector Scripts   |    Shell    |
+|------------------+-----------------+----------------------+-------------|
+|   E-mail User    |    Web User     |      Detector        |   Command   |
+|-------------------------------------------------------------------------|
+|                         Roundup Database Layer                          |
+|-------------------------------------------------------------------------|
+|                          Hyperdatabase Layer                            |
+|-------------------------------------------------------------------------|
+|                             Storage Layer                               |
+-------------------------------------------------------------------------
+The colourful parts of the cake are part of our system;
+the faint grey parts of the cake are external components.
+I will now proceed to forgo all table manners and
+eat from the bottom of the cake to the top.  You may want
+to stand back a bit so you don't get covered in crumbs.
+Hyperdatabase
+-------------
+The lowest-level component to be implemented is the hyperdatabase.
+The hyperdatabase is intended to be
+a flexible data store that can hold configurable data in
+records which we call items.
+The hyperdatabase is implemented on top of the storage layer,
+an external module for storing its data.  The storage layer could
+be a third-party RDBMS; for a "batteries-included" distribution,
+implementing the hyperdatabase on the standard bsddb
+module is suggested.
+Dates and Date Arithmetic
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Before we get into the hyperdatabase itself, we need a
+way of handling dates.  The hyperdatabase module provides
+Timestamp objects for
+representing date-and-time stamps and Interval objects for
+representing date-and-time intervals.
+As strings, date-and-time stamps are specified with
+the date in international standard format
+(``yyyy-mm-dd``)
+joined to the time (``hh:mm:ss``)
+by a period "``.``".  Dates in
+this form can be easily compared and are fairly readable
+when printed.  An example of a valid stamp is
+"``2000-06-24.13:03:59``".
+We'll call this the "full date format".  When Timestamp objects are
+printed as strings, they appear in the full date format with
+the time always given in GMT.  The full date format is always
+exactly 19 characters long.
+For user input, some partial forms are also permitted:
+the whole time or just the seconds may be omitted; and the whole date
+may be omitted or just the year may be omitted.  If the time is given,
+the time is interpreted in the user's local time zone.
+The Date constructor takes care of these conversions.
+In the following examples, suppose that ``yyyy`` is the current year,
+``mm`` is the current month, and ``dd`` is the current
+day of the month; and suppose that the user is on Eastern Standard Time.
+-   "2000-04-17" means <Date 2000-04-17.00:00:00>
+-   "01-25" means <Date yyyy-01-25.00:00:00>
+-   "2000-04-17.03:45" means <Date 2000-04-17.08:45:00>
+-   "08-13.22:13" means <Date yyyy-08-14.03:13:00>
+-   "11-07.09:32:43" means <Date yyyy-11-07.14:32:43>
+-   "14:25" means
+-   <Date yyyy-mm-dd.19:25:00>
+-   "8:47:11" means
+-   <Date yyyy-mm-dd.13:47:11>
+-   the special date "." means "right now"
+Date intervals are specified using the suffixes
+"y", "m", and "d".  The suffix "w" (for "week") means 7 days.
+Time intervals are specified in hh:mm:ss format (the seconds
+may be omitted, but the hours and minutes may not).
+-   "3y" means three years
+-   "2y 1m" means two years and one month
+-   "1m 25d" means one month and 25 days
+-   "2w 3d" means two weeks and three days
+-   "1d 2:50" means one day, two hours, and 50 minutes
+-   "14:00" means 14 hours
+-   "0:04:33" means four minutes and 33 seconds
+The Date class should understand simple date expressions of the form
+*stamp* ``+`` *interval* and *stamp* ``-`` *interval*.
+When adding or subtracting intervals involving months or years, the
+components are handled separately.  For example, when evaluating
+"``2000-06-25 + 1m 10d``", we first add one month to
+get 2000-07-25, then add 10 days to get
+2000-08-04 (rather than trying to decide whether
+1m 10d means 38 or 40 or 41 days).
+Here is an outline of the Date and Interval classes::
+class Date:
+def __init__(self, spec, offset):
+"""Construct a date given a specification and a time zone offset.
+'spec' is a full date or a partial form, with an optional
+added or subtracted interval.  'offset' is the local time
+zone offset from GMT in hours.
+"""
+def __add__(self, interval):
+"""Add an interval to this date to produce another date."""
+def __sub__(self, interval):
+"""Subtract an interval from this date to produce another date."""
+def __cmp__(self, other):
+"""Compare this date to another date."""
+def __str__(self):
+"""Return this date as a string in the yyyy-mm-dd.hh:mm:ss format."""
+def local(self, offset):
+"""Return this date as yyyy-mm-dd.hh:mm:ss in a local time zone."""
+class Interval:
+def __init__(self, spec):
+"""Construct an interval given a specification."""
+def __cmp__(self, other):
+"""Compare this interval to another interval."""
+def __str__(self):
+"""Return this interval as a string."""
+Here are some examples of how these classes would behave in practice.
+For the following examples, assume that we are on Eastern Standard
+Time and the current local time is 19:34:02 on 25 June 2000::
+>>> Date(".")
+<Date 2000-06-26.00:34:02>
+>>> _.local(-5)
+"2000-06-25.19:34:02"
+>>> Date(". + 2d")
+<Date 2000-06-28.00:34:02>
+>>> Date("1997-04-17", -5)
+<Date 1997-04-17.00:00:00>
+>>> Date("01-25", -5)
+<Date 2000-01-25.00:00:00>
+>>> Date("08-13.22:13", -5)
+<Date 2000-08-14.03:13:00>
+>>> Date("14:25", -5)
+<Date 2000-06-25.19:25:00>
+>>> Interval("  3w  1  d  2:00")
+<Interval 22d 2:00>
+>>> Date(". + 2d") - Interval("3w")
+<Date 2000-06-07.00:34:02>
+Items and Classes
+~~~~~~~~~~~~~~~~~
+Items contain data in properties.  To Python, these
+properties are presented as the key-value pairs of a dictionary.
+Each item belongs to a class which defines the names
+and types of its properties.  The database permits the creation
+and modification of classes as well as items.
+Identifiers and Designators
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Each item has a numeric identifier which is unique among
+items in its class.  The items are numbered sequentially
+within each class in order of creation, starting from 1.
+The designator
+for an item is a way to identify an item in the database, and
+consists of the name of the item's class concatenated with
+the item's numeric identifier.
+For example, if "spam" and "eggs" are classes, the first
+item created in class "spam" has id 1 and designator "spam1".
+The first item created in class "eggs" also has id 1 but has
+the distinct designator "eggs1".  Item designators are
+conventionally enclosed in square brackets when mentioned
+in plain text.  This permits a casual mention of, say,
+"[patch37]" in an e-mail message to be turned into an active
+hyperlink.
+Property Names and Types
+~~~~~~~~~~~~~~~~~~~~~~~~
+Property names must begin with a letter.
+A property may be one of five basic types:
+- String properties are for storing arbitrary-length strings.
+- Boolean properties are for storing true/false, or yes/no values.
+- Number properties are for storing numeric values.
+- Date properties store date-and-time stamps.
+Their values are Timestamp objects.
+- A Link property refers to a single other item
+selected from a specified class.  The class is part of the property;
+the value is an integer, the id of the chosen item.
+- A Multilink property refers to possibly many items
+in a specified class.  The value is a list of integers.
+*None* is also a permitted value for any of these property
+types.  An attempt to store None into a Multilink property stores an empty list.
+A property that is not specified will return as None from a *get*
+operation.
+Hyperdb Interface Specification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+TODO: replace the Interface Specifications with links to the pydoc
+The hyperdb module provides property objects to designate
+the different kinds of properties.  These objects are used when
+specifying what properties belong in classes::
+class String:
+def __init__(self, indexme='no'):
+"""An object designating a String property."""
+class Boolean:
+def __init__(self):
+"""An object designating a Boolean property."""
+class Number:
+def __init__(self):
+"""An object designating a Number property."""
+class Date:
+def __init__(self):
+"""An object designating a Date property."""
+class Link:
+def __init__(self, classname, do_journal='yes'):
+"""An object designating a Link property that links to
+items in a specified class.
+If the do_journal argument is not 'yes' then changes to
+the property are not journalled in the linked item.
+"""
+class Multilink:
+def __init__(self, classname, do_journal='yes'):
+"""An object designating a Multilink property that links
+to items in a specified class.
+If the do_journal argument is not 'yes' then changes to
+the property are not journalled in the linked item(s).
+"""
+Here is the interface provided by the hyperdatabase::
+class Database:
+"""A database for storing records containing flexible data types."""
+def __init__(self, config, journaltag=None):
+"""Open a hyperdatabase given a specifier to some storage.
+The 'storagelocator' is obtained from config.DATABASE.
+The meaning of 'storagelocator' depends on the particular
+implementation of the hyperdatabase.  It could be a file name,
+a directory path, a socket descriptor for a connection to a
+database over the network, etc.
+The 'journaltag' is a token that will be attached to the journal
+entries for any edits done on the database.  If 'journaltag' is
+None, the database is opened in read-only mode: the Class.create(),
+Class.set(), and Class.retire() methods are disabled.
+"""
+def __getattr__(self, classname):
+"""A convenient way of calling self.getclass(classname)."""
+def getclasses(self):
+"""Return a list of the names of all existing classes."""
+def getclass(self, classname):
+"""Get the Class object representing a particular class.
+If 'classname' is not a valid class name, a KeyError is raised.
+"""
+class Class:
+"""The handle to a particular class of items in a hyperdatabase."""
+def __init__(self, db, classname, **properties):
+"""Create a new class with a given name and property specification.
+'classname' must not collide with the name of an existing class,
+or a ValueError is raised.  The keyword arguments in 'properties'
+must map names to property objects, or a TypeError is raised.
+"""
+# Editing items:
+def create(self, **propvalues):
+"""Create a new item of this class and return its id.
+The keyword arguments in 'propvalues' map property names to values.
+The values of arguments must be acceptable for the types of their
+corresponding properties or a TypeError is raised.  If this class
+has a key property, it must be present and its value must not
+collide with other key strings or a ValueError is raised.  Any other
+properties on this class that are missing from the 'propvalues'
+dictionary are set to None.  If an id in a link or multilink
+property does not refer to a valid item, an IndexError is raised.
+"""
+def get(self, itemid, propname):
+"""Get the value of a property on an existing item of this class.
+'itemid' must be the id of an existing item of this class or an
+IndexError is raised.  'propname' must be the name of a property
+of this class or a KeyError is raised.
+"""
+def set(self, itemid, **propvalues):
+"""Modify a property on an existing item of this class.
+'itemid' must be the id of an existing item of this class or an
+IndexError is raised.  Each key in 'propvalues' must be the name
+of a property of this class or a KeyError is raised.  All values
+in 'propvalues' must be acceptable types for their corresponding
+properties or a TypeError is raised.  If the value of the key
+property is set, it must not collide with other key strings or a
+ValueError is raised.  If the value of a Link or Multilink
+property contains an invalid item id, a ValueError is raised.
+"""
+def retire(self, itemid):
+"""Retire an item.
+The properties on the item remain available from the get() method,
+and the item's id is never reused.  Retired items are not returned
+by the find(), list(), or lookup() methods, and other items may
+reuse the values of their key properties.
+"""
+def history(self, itemid):
+"""Retrieve the journal of edits on a particular item.
+'itemid' must be the id of an existing item of this class or an
+IndexError is raised.
+The returned list contains tuples of the form
+(date, tag, action, params)
+'date' is a Timestamp object specifying the time of the change and
+'tag' is the journaltag specified when the database was opened.
+'action' may be:
+'create' or 'set' -- 'params' is a dictionary of property values
+'link' or 'unlink' -- 'params' is (classname, itemid, propname)
+'retire' -- 'params' is None
+"""
+# Locating items:
+def setkey(self, propname):
+"""Select a String property of this class to be the key property.
+'propname' must be the name of a String property of this class or
+None, or a TypeError is raised.  The values of the key property on
+all existing items must be unique or a ValueError is raised.
+"""
+def getkey(self):
+"""Return the name of the key property for this class or None."""
+def lookup(self, keyvalue):
+"""Locate a particular item by its key property and return its id.
+If this class has no key property, a TypeError is raised.  If the
+'keyvalue' matches one of the values for the key property among
+the items in this class, the matching item's id is returned;
+otherwise a KeyError is raised.
+"""
+def find(self, propname, itemid):
+"""Get the ids of items in this class which link to the given items.
+'propspec' consists of keyword args propname={itemid:1,}
+'propname' must be the name of a property in this class, or a
+KeyError is raised.  That property must be a Link or Multilink
+property, or a TypeError is raised.
+Any item in this class whose 'propname' property links to any of the
+itemids will be returned. Used by the full text indexing, which
+knows that "foo" occurs in msg1, msg3 and file7, so we have hits
+on these issues:
+db.issue.find(messages={'1':1,'3':1}, files={'7':1})
+"""
+def filter(self, search_matches, filterspec, sort, group):
+''' Return a list of the ids of the active items in this class that
+match the 'filter' spec, sorted by the group spec and then the
+sort spec.
+'''
+def list(self):
+"""Return a list of the ids of the active items in this class."""
+def count(self):
+"""Get the number of items in this class.
+If the returned integer is 'numitems', the ids of all the items
+in this class run from 1 to numitems, and numitems+1 will be the
+id of the next item to be created in this class.
+"""
+# Manipulating properties:
+def getprops(self):
+"""Return a dictionary mapping property names to property objects."""
+def addprop(self, **properties):
+"""Add properties to this class.
+The keyword arguments in 'properties' must map names to property
+objects, or a TypeError is raised.  None of the keys in 'properties'
+may collide with the names of existing properties, or a ValueError
+is raised before any properties have been added.
+"""
+def getitem(self, itemid, cache=1):
+''' Return a Item convenience wrapper for the item.
+'itemid' must be the id of an existing item of this class or an
+IndexError is raised.
+'cache' indicates whether the transaction cache should be queried
+for the item. If the item has been modified and you need to
+determine what its values prior to modification are, you need to
+set cache=0.
+'''
+class Item:
+''' A convenience wrapper for the given item. It provides a mapping
+interface to a single item's properties
+'''
+Hyperdatabase Implementations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Hyperdatabase implementations exist to create the interface described in the
+`hyperdb interface specification`_
+over an existing storage mechanism. Examples are relational databases,
+\*dbm key-value databases, and so on.
+Several implementations are provided - they belong in the roundup.backends
+package.
+Application Example
+~~~~~~~~~~~~~~~~~~~
+Here is an example of how the hyperdatabase module would work in practice::
+>>> import hyperdb
+>>> db = hyperdb.Database("foo.db", "ping")
+>>> db
+<hyperdb.Database "foo.db" opened by "ping">
+>>> hyperdb.Class(db, "status", name=hyperdb.String())
+<hyperdb.Class "status">
+>>> _.setkey("name")
+>>> db.status.create(name="unread")
+1
+>>> db.status.create(name="in-progress")
+2
+>>> db.status.create(name="testing")
+3
+>>> db.status.create(name="resolved")
+4
+>>> db.status.count()
+4
+>>> db.status.list()
+[1, 2, 3, 4]
+>>> db.status.lookup("in-progress")
+2
+>>> db.status.retire(3)
+>>> db.status.list()
+[1, 2, 4]
+>>> hyperdb.Class(db, "issue", title=hyperdb.String(), status=hyperdb.Link("status"))
+<hyperdb.Class "issue">
+>>> db.issue.create(title="spam", status=1)
+1
+>>> db.issue.create(title="eggs", status=2)
+2
+>>> db.issue.create(title="ham", status=4)
+3
+>>> db.issue.create(title="arguments", status=2)
+4
+>>> db.issue.create(title="abuse", status=1)
+5
+>>> hyperdb.Class(db, "user", username=hyperdb.Key(), password=hyperdb.String())
+<hyperdb.Class "user">
+>>> db.issue.addprop(fixer=hyperdb.Link("user"))
+>>> db.issue.getprops()
+{"title": <hyperdb.String>, "status": <hyperdb.Link to "status">,
+"user": <hyperdb.Link to "user">}
+>>> db.issue.set(5, status=2)
+>>> db.issue.get(5, "status")
+2
+>>> db.status.get(2, "name")
+"in-progress"
+>>> db.issue.get(5, "title")
+"abuse"
+>>> db.issue.find("status", db.status.lookup("in-progress"))
+[2, 4, 5]
+>>> db.issue.history(5)
+[(<Date 2000-06-28.19:09:43>, "ping", "create", {"title": "abuse", "status": 1}),
+(<Date 2000-06-28.19:11:04>, "ping", "set", {"status": 2})]
+>>> db.status.history(1)
+[(<Date 2000-06-28.19:09:43>, "ping", "link", ("issue", 5, "status")),
+(<Date 2000-06-28.19:11:04>, "ping", "unlink", ("issue", 5, "status"))]
+>>> db.status.history(2)
+[(<Date 2000-06-28.19:11:04>, "ping", "link", ("issue", 5, "status"))]
+For the purposes of journalling, when a Multilink property is
+set to a new list of items, the hyperdatabase compares the old
+list to the new list.
+The journal records "unlink" events for all the items that appear
+in the old list but not the new list,
+and "link" events for
+all the items that appear in the new list but not in the old list.
+Roundup Database
+----------------
+The Roundup database layer is implemented on top of the
+hyperdatabase and mediates calls to the database.
+Some of the classes in the Roundup database are considered
+issue classes.
+The Roundup database layer adds detectors and user items,
+and on issues it provides mail spools, nosy lists, and superseders.
+Reserved Classes
+~~~~~~~~~~~~~~~~
+Internal to this layer we reserve three special classes
+of items that are not issues.
+Users
+"""""
+Users are stored in the hyperdatabase as items of
+class "user".  The "user" class has the definition::
+hyperdb.Class(db, "user", username=hyperdb.String(),
+password=hyperdb.String(),
+address=hyperdb.String())
+db.user.setkey("username")
+Messages
+""""""""
+E-mail messages are represented by hyperdatabase items of class "msg".
+The actual text content of the messages is stored in separate files.
+(There's no advantage to be gained by stuffing them into the
+hyperdatabase, and if messages are stored in ordinary text files,
+they can be grepped from the command line.)  The text of a message is
+saved in a file named after the message item designator (e.g. "msg23")
+for the sake of the command interface (see below).  Attachments are
+stored separately and associated with "file" items.
+The "msg" class has the definition::
+hyperdb.Class(db, "msg", author=hyperdb.Link("user"),
+recipients=hyperdb.Multilink("user"),
+date=hyperdb.Date(),
+summary=hyperdb.String(),
+files=hyperdb.Multilink("file"))
+The "author" property indicates the author of the message
+(a "user" item must exist in the hyperdatabase for any messages
+that are stored in the system).
+The "summary" property contains a summary of the message for display
+in a message index.
+Files
+"""""
+Submitted files are represented by hyperdatabase
+items of class "file".  Like e-mail messages, the file content
+is stored in files outside the database,
+named after the file item designator (e.g. "file17").
+The "file" class has the definition::
+hyperdb.Class(db, "file", user=hyperdb.Link("user"),
+name=hyperdb.String(),
+type=hyperdb.String())
+The "user" property indicates the user who submitted the
+file, the "name" property holds the original name of the file,
+and the "type" property holds the MIME type of the file as received.
+Issue Classes
+~~~~~~~~~~~~~
+All issues have the following standard properties:
+=========== ==========================
+Property    Definition
+=========== ==========================
+title       hyperdb.String()
+messages    hyperdb.Multilink("msg")
+files       hyperdb.Multilink("file")
+nosy        hyperdb.Multilink("user")
+superseder  hyperdb.Multilink("issue")
+=========== ==========================
+Also, two Date properties named "creation" and "activity" are
+fabricated by the Roundup database layer.  By "fabricated" we
+mean that no such properties are actually stored in the
+hyperdatabase, but when properties on issues are requested, the
+"creation" and "activity" properties are made available.
+The value of the "creation" property is the date when an issue was
+created, and the value of the "activity" property is the
+date when any property on the issue was last edited (equivalently,
+these are the dates on the first and last records in the issue's journal).
+Roundupdb Interface Specification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The interface to a Roundup database delegates most method
+calls to the hyperdatabase, except for the following
+changes and additional methods::
+class Database:
+def getuid(self):
+"""Return the id of the "user" item associated with the user
+that owns this connection to the hyperdatabase."""
+class Class:
+# Overridden methods:
+def create(self, **propvalues):
+def set(self, **propvalues):
+def retire(self, itemid):
+"""These operations trigger detectors and can be vetoed.  Attempts
+to modify the "creation" or "activity" properties cause a KeyError.
+"""
+# New methods:
+def audit(self, event, detector):
+def react(self, event, detector):
+"""Register a detector (see below for more details)."""
+class IssueClass(Class):
+# Overridden methods:
+def __init__(self, db, classname, **properties):
+"""The newly-created class automatically includes the "messages",
+"files", "nosy", and "superseder" properties.  If the 'properties'
+dictionary attempts to specify any of these properties or a
+"creation" or "activity" property, a ValueError is raised."""
+def get(self, itemid, propname):
+def getprops(self):
+"""In addition to the actual properties on the item, these
+methods provide the "creation" and "activity" properties."""
+# New methods:
+def addmessage(self, itemid, summary, text):
+"""Add a message to an issue's mail spool.
+A new "msg" item is constructed using the current date, the
+user that owns the database connection as the author, and
+the specified summary text.  The "files" and "recipients"
+fields are left empty.  The given text is saved as the body
+of the message and the item is appended to the "messages"
+field of the specified issue.
+"""
+def sendmessage(self, itemid, msgid):
+"""Send a message to the members of an issue's nosy list.
+The message is sent only to users on the nosy list who are not
+already on the "recipients" list for the message.  These users
+are then added to the message's "recipients" list.
+"""
+Default Schema
+~~~~~~~~~~~~~~
+The default schema included with Roundup turns it into a
+typical software bug tracker.  The database is set up like this::
+pri = Class(db, "priority", name=hyperdb.String(), order=hyperdb.String())
+pri.setkey("name")
+pri.create(name="critical", order="1")
+pri.create(name="urgent", order="2")
+pri.create(name="bug", order="3")
+pri.create(name="feature", order="4")
+pri.create(name="wish", order="5")
+stat = Class(db, "status", name=hyperdb.String(), order=hyperdb.String())
+stat.setkey("name")
+stat.create(name="unread", order="1")
+stat.create(name="deferred", order="2")
+stat.create(name="chatting", order="3")
+stat.create(name="need-eg", order="4")
+stat.create(name="in-progress", order="5")
+stat.create(name="testing", order="6")
+stat.create(name="done-cbb", order="7")
+stat.create(name="resolved", order="8")
+Class(db, "keyword", name=hyperdb.String())
+Class(db, "issue", fixer=hyperdb.Multilink("user"),
+topic=hyperdb.Multilink("keyword"),
+priority=hyperdb.Link("priority"),
+status=hyperdb.Link("status"))
+(The "order" property hasn't been explained yet.  It
+gets used by the Web user interface for sorting.)
+The above isn't as pretty-looking as the schema specification
+in the first-stage submission, but it could be made just as easy
+with the addition of a convenience function like Choice
+for setting up the "priority" and "status" classes::
+def Choice(name, *options):
+cl = Class(db, name, name=hyperdb.String(), order=hyperdb.String())
+for i in range(len(options)):
+cl.create(name=option[i], order=i)
+return hyperdb.Link(name)
+Detector Interface
+------------------
+Detectors are Python functions that are triggered on certain
+kinds of events.  The definitions of the
+functions live in Python modules placed in a directory set aside
+for this purpose.  Importing the Roundup database module also
+imports all the modules in this directory, and the ``init()``
+function of each module is called when a database is opened to
+provide it a chance to register its detectors.
+There are two kinds of detectors:
+1. an auditor is triggered just before modifying an item
+2. a reactor is triggered just after an item has been modified
+When the Roundup database is about to perform a
+``create()``, ``set()``, or ``retire()``
+operation, it first calls any *auditors* that
+have been registered for that operation on that class.
+Any auditor may raise a *Reject* exception
+to abort the operation.
+If none of the auditors raises an exception, the database
+proceeds to carry out the operation.  After it's done, it
+then calls all of the *reactors* that have been registered
+for the operation.
+Detector Interface Specification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The ``audit()`` and ``react()`` methods
+register detectors on a given class of items::
+class Class:
+def audit(self, event, detector):
+"""Register an auditor on this class.
+'event' should be one of "create", "set", or "retire".
+'detector' should be a function accepting four arguments.
+"""
+def react(self, event, detector):
+"""Register a reactor on this class.
+'event' should be one of "create", "set", or "retire".
+'detector' should be a function accepting four arguments.
+"""
+Auditors are called with the arguments::
+audit(db, cl, itemid, newdata)
+where ``db`` is the database, ``cl`` is an
+instance of Class or IssueClass within the database, and ``newdata``
+is a dictionary mapping property names to values.
+For a ``create()``
+operation, the ``itemid`` argument is None and newdata
+contains all of the initial property values with which the item
+is about to be created.
+For a ``set()`` operation, newdata
+contains only the names and values of properties that are about
+to be changed.
+For a ``retire()`` operation, newdata is None.
+Reactors are called with the arguments::
+react(db, cl, itemid, olddata)
+where ``db`` is the database, ``cl`` is an
+instance of Class or IssueClass within the database, and ``olddata``
+is a dictionary mapping property names to values.
+For a ``create()``
+operation, the ``itemid`` argument is the id of the
+newly-created item and ``olddata`` is None.
+For a ``set()`` operation, ``olddata``
+contains the names and previous values of properties that were changed.
+For a ``retire()`` operation, ``itemid`` is the
+id of the retired item and ``olddata`` is None.
+Detector Example
+~~~~~~~~~~~~~~~~
+Here is an example of detectors written for a hypothetical
+project-management application, where users can signal approval
+of a project by adding themselves to an "approvals" list, and
+a project proceeds when it has three approvals::
+# Permit users only to add themselves to the "approvals" list.
+def check_approvals(db, cl, id, newdata):
+if newdata.has_key("approvals"):
+if cl.get(id, "status") == db.status.lookup("approved"):
+raise Reject, "You can't modify the approvals list " \
+"for a project that has already been approved."
+old = cl.get(id, "approvals")
+new = newdata["approvals"]
+for uid in old:
+if uid not in new and uid != db.getuid():
+raise Reject, "You can't remove other users from the "
+"approvals list; you can only remove yourself."
+for uid in new:
+if uid not in old and uid != db.getuid():
+raise Reject, "You can't add other users to the approvals "
+"list; you can only add yourself."
+# When three people have approved a project, change its
+# status from "pending" to "approved".
+def approve_project(db, cl, id, olddata):
+if olddata.has_key("approvals") and len(cl.get(id, "approvals")) == 3:
+if cl.get(id, "status") == db.status.lookup("pending"):
+cl.set(id, status=db.status.lookup("approved"))
+def init(db):
+db.project.audit("set", check_approval)
+db.project.react("set", approve_project)
+Here is another example of a detector that can allow or prevent
+the creation of new items.  In this scenario, patches for a software
+project are submitted by sending in e-mail with an attached file,
+and we want to ensure that there are text/plain attachments on
+the message.  The maintainer of the package can then apply the
+patch by setting its status to "applied"::
+# Only accept attempts to create new patches that come with patch files.
+def check_new_patch(db, cl, id, newdata):
+if not newdata["files"]:
+raise Reject, "You can't submit a new patch without " \
+"attaching a patch file."
+for fileid in newdata["files"]:
+if db.file.get(fileid, "type") != "text/plain":
+raise Reject, "Submitted patch files must be text/plain."
+# When the status is changed from "approved" to "applied", apply the patch.
+def apply_patch(db, cl, id, olddata):
+if cl.get(id, "status") == db.status.lookup("applied") and \
+olddata["status"] == db.status.lookup("approved"):
+# ...apply the patch...
+def init(db):
+db.patch.audit("create", check_new_patch)
+db.patch.react("set", apply_patch)
+Command Interface
+-----------------
+The command interface is a very simple and minimal interface,
+intended only for quick searches and checks from the shell prompt.
+(Anything more interesting can simply be written in Python using
+the Roundup database module.)
+Command Interface Specification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+A single command, roundup, provides basic access to
+the hyperdatabase from the command line::
+roundup-admin help
+roundup-admin get [-list] designator[, designator,...] propname
+roundup-admin set designator[, designator,...] propname=value ...
+roundup-admin find [-list] classname propname=value ...
+See ``roundup-admin help commands`` for a complete list of commands.
+Property values are represented as strings in command arguments
+and in the printed results:
+- Strings are, well, strings.
+- Numbers are displayed the same as strings.
+- Booleans are displayed as 'Yes' or 'No'.
+- Date values are printed in the full date format in the local
+time zone, and accepted in the full format or any of the partial
+formats explained above.
+- Link values are printed as item designators.  When given as
+an argument, item designators and key strings are both accepted.
+- Multilink values are printed as lists of item designators
+joined by commas.  When given as an argument, item designators
+and key strings are both accepted; an empty string, a single item,
+or a list of items joined by commas is accepted.
+When multiple items are specified to the
+roundup get or roundup set
+commands, the specified properties are retrieved or set
+on all the listed items.
+When multiple results are returned by the roundup get
+or roundup find commands, they are printed one per
+line (default) or joined by commas (with the -list) option.
+Usage Example
+~~~~~~~~~~~~~
+To find all messages regarding in-progress issues that
+contain the word "spam", for example, you could execute the
+following command from the directory where the database
+dumps its files::
+shell% for issue in `roundup find issue status=in-progress`; do
+> grep -l spam `roundup get $issue messages`
+> done
+msg23
+msg49
+msg50
+msg61
+shell%
+Or, using the -list option, this can be written as a single command::
+shell% grep -l spam `roundup get \
+\`roundup find -list issue status=in-progress\` messages`
+msg23
+msg49
+msg50
+msg61
+shell%
+E-mail User Interface
+---------------------
+The Roundup system must be assigned an e-mail address
+at which to receive mail.  Messages should be piped to
+the Roundup mail-handling script by the mail delivery
+system (e.g. using an alias beginning with "|" for sendmail).
+Message Processing
+~~~~~~~~~~~~~~~~~~
+Incoming messages are examined for multiple parts.
+In a multipart/mixed message or part, each subpart is
+extracted and examined.  In a multipart/alternative
+message or part, we look for a text/plain subpart and
+ignore the other parts.  The text/plain subparts are
+assembled to form the textual body of the message, to
+be stored in the file associated with a "msg" class item.
+Any parts of other types are each stored in separate
+files and given "file" class items that are linked to
+the "msg" item.
+The "summary" property on message items is taken from
+the first non-quoting section in the message body.
+The message body is divided into sections by blank lines.
+Sections where the second and all subsequent lines begin
+with a ">" or "|" character are considered "quoting
+sections".  The first line of the first non-quoting
+section becomes the summary of the message.
+All of the addresses in the To: and Cc: headers of the
+incoming message are looked up among the user items, and
+the corresponding users are placed in the "recipients"
+property on the new "msg" item.  The address in the From:
+header similarly determines the "author" property of the
+new "msg" item.
+The default handling for
+addresses that don't have corresponding users is to create
+new users with no passwords and a username equal to the
+address.  (The web interface does not permit logins for
+users with no passwords.)  If we prefer to reject mail from
+outside sources, we can simply register an auditor on the
+"user" class that prevents the creation of user items with
+no passwords.
+The subject line of the incoming message is examined to
+determine whether the message is an attempt to create a new
+issue or to discuss an existing issue.  A designator enclosed
+in square brackets is sought as the first thing on the
+subject line (after skipping any "Fwd:" or "Re:" prefixes).
+If an issue designator (class name and id number) is found
+there, the newly created "msg" item is added to the "messages"
+property for that issue, and any new "file" items are added to
+the "files" property for the issue.
+If just an issue class name is found there, we attempt to
+create a new issue of that class with its "messages" property
+initialized to contain the new "msg" item and its "files"
+property initialized to contain any new "file" items.
+Both cases may trigger detectors (in the first case we
+are calling the set() method to add the message to the
+issue's spool; in the second case we are calling the
+create() method to create a new item).  If an auditor
+raises an exception, the original message is bounced back to
+the sender with the explanatory message given in the exception.
+Nosy Lists
+~~~~~~~~~~
+A standard detector is provided that watches for additions
+to the "messages" property.  When a new message is added, the
+detector sends it to all the users on the "nosy" list for the
+issue that are not already on the "recipients" list of the
+message.  Those users are then appended to the "recipients"
+property on the message, so multiple copies of a message
+are never sent to the same user.  The journal recorded by
+the hyperdatabase on the "recipients" property then provides
+a log of when the message was sent to whom.
+Setting Properties
+~~~~~~~~~~~~~~~~~~
+The e-mail interface also provides a simple way to set
+properties on issues.  At the end of the subject line,
+``propname=value`` pairs can be
+specified in square brackets, using the same conventions
+as for the roundup ``set`` shell command.
+Web User Interface
+------------------
+The web interface is provided by a CGI script that can be
+run under any web server.  A simple web server can easily be
+built on the standard CGIHTTPServer module, and
+should also be included in the distribution for quick
+out-of-the-box deployment.
+The user interface is constructed from a number of template
+files containing mostly HTML.  Among the HTML tags in templates
+are interspersed some nonstandard tags, which we use as
+placeholders to be replaced by properties and their values.
+Views and View Specifiers
+~~~~~~~~~~~~~~~~~~~~~~~~~
+There are two main kinds of views: *index* views and *issue* views.
+An index view displays a list of issues of a particular class,
+optionally sorted and filtered as requested.  An issue view
+presents the properties of a particular issue for editing
+and displays the message spool for the issue.
+A view specifier is a string that specifies
+all the options needed to construct a particular view.
+It goes after the URL to the Roundup CGI script or the
+web server to form the complete URL to a view.  When the
+result of selecting a link or submitting a form takes
+the user to a new view, the Web browser should be redirected
+to a canonical location containing a complete view specifier
+so that the view can be bookmarked.
+Displaying Properties
+~~~~~~~~~~~~~~~~~~~~~
+Properties appear in the user interface in three contexts:
+in indices, in editors, and as search filters.  For each type of
+property, there are several display possibilities.  For example,
+in an index view, a string property may just be printed as
+a plain string, but in an editor view, that property should
+be displayed in an editable field.
+The display of a property is handled by functions in
+the ``cgi.templating`` module.
+Displayer functions are triggered by ``tal:content`` or ``tal:replace``
+tag attributes in templates.  The value of the attribute
+provides an expression for calling the displayer function.
+For example, the occurrence of::
+tal:content="context/status/plain"
+in a template triggers a call to::
+context['status'].plain()
+where the context would be an item of the "issue" class.  The displayer
+functions can accept extra arguments to further specify
+details about the widgets that should be generated.
+Some of the standard displayer functions include:
+========= ====================================================================
+Function  Description
+========= ====================================================================
+plain     display a String property directly;
+display a Date property in a specified time zone with an option
+to omit the time from the date stamp; for a Link or Multilink
+property, display the key strings of the linked items (or the
+ids if the linked class has no key property)
+field     display a property like the plain displayer above, but in a text
+field to be edited
+menu      for a Link property, display a menu of the available choices
+========= ====================================================================
+See the `customisation`_ documentation for the complete list.
+Index Views
+~~~~~~~~~~~
+XXX The following needs to be clearer
+An index view contains two sections: a filter section
+and an index section.
+The filter section provides some widgets for selecting
+which issues appear in the index.  The index section is
+a table of issues.
+Index View Specifiers
+"""""""""""""""""""""
+An index view specifier looks like this (whitespace
+has been added for clarity)::
+/issue?status=unread,in-progress,resolved&
+topic=security,ui&
+:group=priority&
+:sort=-activity&
+:filters=status,topic&
+:columns=title,status,fixer
+The index view is determined by two parts of the
+specifier: the layout part and the filter part.
+The layout part consists of the query parameters that
+begin with colons, and it determines the way that the
+properties of selected items are displayed.
+The filter part consists of all the other query parameters,
+and it determines the criteria by which items
+are selected for display.
+The filter part is interactively manipulated with
+the form widgets displayed in the filter section.  The
+layout part is interactively manipulated by clicking
+on the column headings in the table.
+The filter part selects the union of the
+sets of issues with values matching any specified Link
+properties and the intersection of the sets
+of issues with values matching any specified Multilink
+properties.
+The example specifies an index of "issue" items.
+Only issues with a "status" of either
+"unread" or "in-progres" or "resolved" are displayed,
+and only issues with "topic" values including both
+"security" and "ui" are displayed.  The issues
+are grouped by priority, arranged in ascending order;
+and within groups, sorted by activity, arranged in
+descending order.  The filter section shows filters
+for the "status" and "topic" properties, and the
+table includes columns for the "title", "status", and
+"fixer" properties.
+Associated with each issue class is a default
+layout specifier.  The layout specifier in the above
+example is the default layout to be provided with
+the default bug-tracker schema described above in
+section 4.4.
+Index Section
+"""""""""""""
+The template for an index section describes one row of
+the index table.
+Fragments enclosed in ``<property>...</property>``
+tags are included or omitted depending on whether the
+view specifier requests a column for a particular property.
+The table cells should contain <display> tags
+to display the values of the issue's properties.
+Here's a simple example of an index template::
+<tr>
+<td tal:condition="request/show/title" tal:content="contex/title"></td>
+<td tal:condition="request/show/status" tal:content="contex/status"></td>
+<td tal:condition="request/show/fixer" tal:content="contex/fixer"></td>
+</tr>
+Sorting
+"""""""
+String and Date values are sorted in the natural way.
+Link properties are sorted according to the value of the
+"order" property on the linked items if it is present; or
+otherwise on the key string of the linked items; or
+finally on the item ids.  Multilink properties are
+sorted according to how many links are present.
+Issue Views
+~~~~~~~~~~~
+An issue view contains an editor section and a spool section.
+At the top of an issue view, links to superseding and superseded
+issues are always displayed.
+Issue View Specifiers
+"""""""""""""""""""""
+An issue view specifier is simply the issue's designator::
+/patch23
+Editor Section
+""""""""""""""
+The editor section is generated from a template
+containing <display> tags to insert
+the appropriate widgets for editing properties.
+Here's an example of a basic editor template::
+<table>
+<tr>
+<td colspan=2 tal:content="python:context.title.field(size='60')"></td>
+</tr>
+<tr>
+<td tal:content="context/fixer/field"></td>
+<td tal:content="context/status/menu"></td>
+</tr>
+<tr>
+<td tal:content="context/nosy/field"></td>
+<td tal:content="context/priority/menu"></td>
+</tr>
+<tr>
+<td colspan=2>
+<textarea name=":note" rows=5 cols=60></textarea>
+</td>
+</tr>
+</table>
+As shown in the example, the editor template can also include a ":note" field,
+which is a text area for entering a note to go along with a change.
+When a change is submitted, the system automatically
+generates a message describing the changed properties.
+The message displays all of the property values on the
+issue and indicates which ones have changed.
+An example of such a message might be this::
+title: Polly Parrot is dead
+priority: critical
+status: unread -> in-progress
+fixer: (none)
+keywords: parrot,plumage,perch,nailed,dead
+If a note is given in the ":note" field, the note is
+appended to the description.  The message is then added
+to the issue's message spool (thus triggering the standard
+detector to react by sending out this message to the nosy list).
+Spool Section
+"""""""""""""
+The spool section lists messages in the issue's "messages"
+property.  The index of messages displays the "date", "author",
+and "summary" properties on the message items, and selecting a
+message takes you to its content.
+Access Control
+--------------
+At each point that requires an action to be performed, the security mechanisms
+are asked if the current user has permission. This permission is defined as a
+Permission.
+Individual assignment of Permission to user is unwieldy. The concept of a
+Role, which encompasses several Permissions and may be assigned to many Users,
+is quite well developed in many projects. Roundup will take this path, and
+allow the multiple assignment of Roles to Users, and multiple Permissions to
+Roles. These definitions are not persistent - they're defined when the
+application initialises.
+There will be two levels of Permission. The Class level permissions define
+logical permissions associated with all items of a particular class (or all
+classes). The Item level permissions define logical permissions associated
+with specific items by way of their user-linked properties.
+Access Control Interface Specification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The security module defines::
+class Permission:
+''' Defines a Permission with the attributes
+- name
+- description
+- klass (optional)
+The klass may be unset, indicating that this permission is not
+locked to a particular hyperdb class. There may be multiple
+Permissions for the same name for different classes.
+'''
+class Role:
+''' Defines a Role with the attributes
+- name
+- description
+- permissions
+'''
+class Security:
+def __init__(self, db):
+''' Initialise the permission and role stores, and add in the
+base roles (for admin user).
+'''
+def getPermission(self, permission, classname=None):
+''' Find the Permission matching the name and for the class, if the
+classname is specified.
+Raise ValueError if there is no exact match.
+'''
+def hasPermission(self, permission, userid, classname=None):
+''' Look through all the Roles, and hence Permissions, and see if
+"permission" is there for the specified classname.
+'''
+def hasItemPermission(self, classname, itemid, **propspec):
+''' Check the named properties of the given item to see if the
+userid appears in them. If it does, then the user is granted
+this permission check.
+'propspec' consists of a set of properties and values that
+must be present on the given item for access to be granted.
+If a property is a Link, the value must match the property
+value. If a property is a Multilink, the value must appear
+in the Multilink list.
+'''
+def addPermission(self, **propspec):
+''' Create a new Permission with the properties defined in
+'propspec'
+'''
+def addRole(self, **propspec):
+''' Create a new Role with the properties defined in 'propspec'
+'''
+def addPermissionToRole(self, rolename, permission):
+''' Add the permission to the role's permission list.
+'rolename' is the name of the role to add permission to.
+'''
+Modules such as ``cgi/client.py`` and ``mailgw.py`` define their own
+permissions like so (this example is ``cgi/client.py``)::
+def initialiseSecurity(security):
+''' Create some Permissions and Roles on the security object
+This function is directly invoked by security.Security.__init__()
+as a part of the Security object instantiation.
+'''
+p = security.addPermission(name="Web Registration",
+description="Anonymous users may register through the web")
+security.addToRole('Anonymous', p)
+Detectors may also define roles in their init() function::
+def init(db):
+# register an auditor that checks that a user has the "May Resolve"
+# Permission before allowing them to set an issue status to "resolved"
+db.issue.audit('set', checkresolvedok)
+p = db.security.addPermission(name="May Resolve", klass="issue")
+security.addToRole('Manager', p)
+The tracker dbinit module then has in ``open()``::
+# open the database - it must be modified to init the Security class
+# from security.py as db.security
+db = Database(config, name)
+# add some extra permissions and associate them with roles
+ei = db.security.addPermission(name="Edit", klass="issue",
+description="User is allowed to edit issues")
+db.security.addPermissionToRole('User', ei)
+ai = db.security.addPermission(name="View", klass="issue",
+description="User is allowed to access issues")
+db.security.addPermissionToRole('User', ai)
+In the dbinit ``init()``::
+# create the two default users
+user.create(username="admin", password=Password(adminpw),
+address=config.ADMIN_EMAIL, roles='Admin')
+user.create(username="anonymous", roles='Anonymous')
+Then in the code that matters, calls to ``hasPermission`` and
+``hasItemPermission`` are made to determine if the user has permission
+to perform some action::
+if db.security.hasPermission('issue', 'Edit', userid):
+# all ok
+if db.security.hasItemPermission('issue', itemid, assignedto=userid):
+# all ok
+Code in the core will make use of these methods, as should code in auditors in
+custom templates. The HTML templating may access the access controls through
+the *user* attribute of the *request* variable. It exposes a ``hasPermission()``
+method::
+tal:condition="python:request.user.hasPermission('Edit', 'issue')"
+or, if the *context* is *issue*, then the following is the same::
+tal:condition="python:request.user.hasPermission('Edit')"
+Authentication of Users
+~~~~~~~~~~~~~~~~~~~~~~~
+Users must be authenticated correctly for the above controls to work. This is
+not done in the current mail gateway at all. Use of digital signing of
+messages could alleviate this problem.
+The exact mechanism of registering the digital signature should be flexible,
+with perhaps a level of trust. Users who supply their signature through their
+first message into the tracker should be at a lower level of trust to those
+who supply their signature to an admin for submission to their user details.
+Anonymous Users
+~~~~~~~~~~~~~~~
+The "anonymous" user must always exist, and defines the access permissions for
+anonymous users. Unknown users accessing Roundup through the web or email
+interfaces will be logged in as the "anonymous" user.
+Use Cases
+~~~~~~~~~
+public - end users can submit bugs, request new features, request support
+The Users would be given the default "User" Role which gives "View" and
+"Edit" Permission to the "issue" class.
+developer - developers can fix bugs, implement new features, provide support
+A new Role "Developer" is created with the Permission "Fixer" which is
+checked for in custom auditors that see whether the issue is being
+resolved with a particular resolution ("fixed", "implemented",
+"supported") and allows that resolution only if the permission is
+available.
+manager - approvers/managers can approve new features and signoff bug fixes
+A new Role "Manager" is created with the Permission "Signoff" which is
+checked for in custom auditors that see whether the issue status is being
+changed similar to the developer example.
+admin - administrators can add users and set user's roles
+The existing Role "Admin" has the Permissions "Edit" for all classes
+(including "user") and "Web Roles" which allow the desired actions.
+system - automated request handlers running various report/escalation scripts
+A combination of existing and new Roles, Permissions and auditors could
+be used here.
+privacy - issues that are only visible to some users
+A new property is added to the issue which marks the user or group of
+users who are allowed to view and edit the issue. An auditor will check
+for edit access, and the template user object can check for view access.
+Deployment Scenarios
+--------------------
+The design described above should be general enough
+to permit the use of Roundup for bug tracking, managing
+projects, managing patches, or holding discussions.  By
+using items of multiple types, one could deploy a system
+that maintains requirement specifications, catalogs bugs,
+and manages submitted patches, where patches could be
+linked to the bugs and requirements they address.
+Acknowledgements
+----------------
+My thanks are due to Christy Heyl for
+reviewing and contributing suggestions to this paper
+and motivating me to get it done, and to
+Jesse Vincent, Mark Miller, Christopher Simons,
+Jeff Dunmall, Wayne Gramlich, and Dean Tribble for
+their assistance with the first-round submission.
+Changes to this document
+------------------------
+- Added Boolean and Number types
+- Added section Hyperdatabase Implementations
+- "Item" has been renamed to "Issue" to account for the more specific nature
+of the Class.
+- New Templating
+- Access Controls
+------------------
+Back to `Table of Contents`_
+.. _`Table of Contents`: index.html
+.. _customisation: customizing.html

Mercurial > p > roundup > code

comparison doc/design.txt @ 1356:83f33642d220 maint-0.5