code-1a221cf0758b/doc/FAQ.txt0000644000000000000000000002015614427061170013564 0ustar 00000000000000=========== Roundup FAQ =========== .. contents:: :local: Installation ------------ Living without a mailserver ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Remove the nosy reactor - delete the tracker file ``detectors/nosyreactor.py`` from your tracker home. You do need to configure the ``[mail]`` section of config.ini. Specifically the ``domain`` and ``host`` settings have to have a value. The values don't matter, but the config parser will complain if they are not set. The cgi-bin is very slow! ~~~~~~~~~~~~~~~~~~~~~~~~~ Yep, it sure is. It has to start up Python and load all of the support libraries for *every* request. The solution is to use the built in server (or possibly mod_wsgi or other WSGI support). To make Roundup more seamless with your website, you may place the built in server behind apache and link it into your web tree (see below). How do I put Roundup behind Apache ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We have a project (foo) running on ``tracker.example:8080``. We want ``http://tracker.example/issues`` to use the roundup server, so we set that up on port 8080 on ``tracker.example`` with the ``config.ini`` line:: [tracker] ... web = 'http://tracker.example/issues/' We have a "foo_issues" tracker and we run the server with:: roundup-server -p 8080 issues=/home/roundup/trackers/issues Then, on the Apache machine (eg. redhat 7.3 with apache 1.3), in ``/etc/httpd/conf/httpd.conf`` uncomment:: LoadModule proxy_module modules/libproxy.so and:: AddModule mod_proxy.c Then add:: # roundup stuff (added manually) # proxy through one tracker ProxyPass /issues/ http://tracker.example:8080/issues/ # proxy through all tracker(*) #ProxyPass /roundup/ http://tracker.example:8080/ Then restart Apache. Now Apache will proxy the request on to the roundup-server. Note that if you're proxying multiple trackers, you'll need to use the second ProxyPass rule described above. It will mean that your TRACKER_WEB will change to:: TRACKER_WEB = 'http://tracker.example/roundup/issues/' Once you're done, you can firewall off port 8080 from the rest of the world. Note that in some situations (eg. virtual hosting) you might need to use a more complex rewrite rule instead of the simpler ProxyPass above. The following should be useful as a starting template:: # roundup stuff (added manually) RewriteEngine on # General Roundup RewriteRule ^/roundup$ roundup/ [R] RewriteRule ^/roundup/(.*)$ http://tracker.example:8080/$1 [P,L] # Handle Foo Issues RewriteRule ^/issues$ issues/ [R] RewriteRule ^/issues/(.*)$ http://tracker.example:8080/issues/$1 [P,L] How do I run Roundup through SSL (HTTPS)? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The preferred way of using SSL is to proxy through apache and use its SSL service. See the previous question on how to proxy through apache. The standalone roundup-server now also has SSL support which is still considered experimental. For details refer to the documentation of roundup server, in particular to the generated configuration file generated with :: roundup-server --save-config that describes the needed option in detail. With the standalone server now XMLRPC over SSL works, too. Templates --------- What is that stuff in the tracker html directory? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is the template code that Roundup uses to display the various pages. This is based upon the template markup language in Zope called, oddly enough, "Zope Page Templates". There's documentation in the Roundup customisation_ documentation. For more information have a look at: https://zope.readthedocs.io/en/latest/zopebook/ZPT.html specifically the chapter (10) on "Using Zope Page Templates" and the chapter (13) on "Advanced Page Templates". (Note the link above is for a newer version of Zope, so some of the info may not apply to version 2 of Zope which is used in roundup. The version 2 docs appear to not be available anymore.) But I just want a select/option list for .... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Really easy... edit ``html/issue.item.html``. For ``nosy``, change the line (around line 69) from:: to:: For ``assigned to``, this is already done around line 77:: assignedto menu Great! But now the select/option list is too big ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ That is a little harder (but only a little ;^) Again, edit ``html/issue.item``. For nosy, change line (around line 69) from:: to:: for more information, go and read about Zope Page Templates. I want to link version identifiers from my messages to svn/mercurial/git ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ See: the LinkFormatingExample at: https://wiki.roundup-tracker.org/LinkFormattingExample?highlight=%28local%5C_replace%29 There are examples in the devel and responsive templates. Search for LocalReplace to find the extension file and the change that has to be made to the item template. I can see values in a multilink but not display the items ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Make sure that you have allowed the user to view the labelprop for the multilink class. Look for the setlabelprop explanation in the Roundup customisation_ documentation. Using Roundup ------------- I got an error and I cannot reload it! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you're using Netscape/Mozilla, try holding shift and pressing reload. If you're using IE then install Mozilla and try again ;^) I keep getting logged out ~~~~~~~~~~~~~~~~~~~~~~~~~ Make sure that the ``tracker`` -> ``web`` setting in your tracker's config.ini is set to the URL of the tracker. I'm getting infinite redirects in the browser ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A wrong value for the ``tracker`` -> ``web`` setting may also result in infinite redirects, see https://issues.roundup-tracker.org/issue2537286 How is sorting performed, and why does it seem to fail sometimes? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When we sort items in the hyperdb, we use one of a number of methods, depending on the properties being sorted on: 1. If it's a String, Integer, Number, Date or Interval property, we just sort the scalar value of the property. Strings are sorted case-sensitively. 2. If it's a Link property, we sort by either the linked item's "order" property (if it has one) or the linked item's "id". 3. Mulitlinks sort similar to #2, but we start with the first Multilink list item, and if they're the same, we sort by the second item, and so on. Note that if an "order" property is defined on a Class that is used for sorting, all items of that Class *must* have a value against the "order" property, or sorting will result in random ordering. How do I filter roundup emails? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When roundup emails users, it provides an email header:: X-Roundup-(class)-(property): values for every property that is a link or multilink (usually things set by a dropdown or selection in the web interface). For example, assume you have a category field for issues in your tracker. You respond to issues that fall into the catagories: Network and Security. You will see headers like: X-Roundup-issue-category: Network or X-Roundup-issue-category: Network, Security, Linux Then you can set up a filter looking for X-Roundup-issue-category followed by Network or Security in your mail program. Also for newer (post version 1.6.X) releases there is also a header: X-Roundup-issue-Id: 22 for messages from issue22. Directions on using your mail program are beyond the scope of this FAQ entry. (If you don't see a header for a multilink or link property, look for msg_header_property in the roundup docs. In some cases you have to explicitly enable the header.) .. _`customisation`: customizing.html code-1a221cf0758b/doc/_static/style.css0000644000000000000000000001444514427061170015720 0ustar 00000000000000/* layout*/ /* Set up fluid type scale. Major third for smaller screens and perfect fourth for larger screens. Allow override using --font-level-X. https://set.studio/some-simple-ways-to-make-content-look-good/ */ :root { --_font-level-0: var(--font-level-0, clamp(1rem, 0.96rem + 0.22vw, 1.13rem) ); --_font-level-1: var(--font-level-1, clamp( 1.25 * var(--_font-level-0), 1.16rem + 0.43vw, 1.5rem) ); --_font-level-2: var(--font-level-2, clamp( 1.25 * var(--_font-level-1), 1.41rem + 0.76vw, 2rem) ); --_font-level-3: var(--font-level-3, clamp( 1.25 * var(--_font-level-2), 1.71rem + 1.24vw, 2.66rem) ); --_font-level-4: var(--font-level-4, clamp( 1.25 * var(--_font-level-3), 2.05rem + 1.93vw, 3.55rem) ); } h1, h2, h3, h4, h5, h6 { line-height: 1.25; /* Larger spacing before header and smaller after to make header part of following section */ margin-block-end: 0.3em; } h1 { font-size: var(--_font-level-4); } /* remove whitespace at top of main column */ main h1 { margin-block-start: 0; } h2 { font-size: var(--_font-level-3); } h3 { font-size: var(--_font-level-2); } h4 { font-size: var(--_font-level-1); } /* snug header up to first paragraph of it's section. */ :is(h1,h2,h3,h4,h5,h6) + p { margin-block-start: 0; } body { background-color: #f5f5f5; color: #222; font-family: sans-serif, Arial, Helvetica; font-size: var(--_font-level-0); margin:0; padding: 0 3em 0 14em; } /* -14em is size of table of contents/nav */ body > .header { margin: 0 0 0 -14em;} body > .header div.label { font-size: 2em; font-weight: bold; margin: 0.67em 0 0.67em 1em;} body > .footer { margin: 1em 0 1em -14em; clear:both;} body > .navigation { margin-left: -14em; margin-inline-start: -14em; width: 14em; float: left; } body > .content { margin: 0; width: 100%; } body > .header > #searchbox { position: absolute; right: 1em; top: 1em;} body > .content > #subnav { position: absolute; right: 1.5em; top: 6em;} /* limit width of main column to 65 characters. On wider screens can be 100+ chars: too wide */ main { max-width: 65ch; line-height: calc(1.5 * 1em); } /* style */ :link { color: #bb0000; text-decoration: none;} :link:hover {text-decoration: underline;} :visited { color: #770000; text-decoration: none;} a.toc-backref { color: #000000; } .header h1 { margin-left: 1em; } .menu { padding: 0; margin-right: 1em;} .menu ul { padding: 0; margin: 0; } .menu li { margin: 5pt 0; } .menu > ul > li > * { display: block; padding: 2pt 2pt 2pt 10pt; border: solid thin #dadada; background-color:#ffffff; } .menu > ul > li.current > * { background-color:#dddddd; } .menu ul li:first-child { margin-top:0;} .menu ul { list-style-type:none;} /* sub-menus are indented */ .menu > ul > li > ul, .menu > ul > li.current > ul { border:none; background-color: inherit; } .menu ul ul { margin-left: 1em; font-size: smaller; } /* sub-menu items draw a separator */ .menu ul ul > li { margin: 0; padding: 0; border: none; border-top: solid thin #dadada; background-color: inherit; } .menu ul ul > li:first-child { border-top: none; } .menu ul li.toctree-l2.current { background-color: #dddddd; pading-block: 0.25em; } /* related */ div.related { width: 100%; font-size: 90%; } div.related-top { border-bottom: solid thin #dadada;} div.related-bottom { border-top: solid thin #dadada;} div.related ul { margin: 0; padding: 0 0 0 10px; list-style: none; } div.related li { display: inline;} div.related li.right { float: right; margin-right: 5px; } .footer { font-size: small; text-align: center; color: black; } .footer img { vertical-align: middle; } .content { padding: 1em; border: solid thin #dadada; background-color: #f7f7f7; } /* This is a little hack to inject a 'news' block into the title page without having to set up a custom directive. */ #roundup-issue-tracker .note { float: right; width: auto; border: solid thin #dadada; background-color:#f5f5f5; padding: 1em; margin: 1em; } #roundup-issue-tracker .note .admonition-title { display: none; } table { border-collapse: collapse; border-spacing: 1px; background-color: #fafafa; } table:has(caption) { margin-block: 1em; } table caption { font-weight: bold; font-size: smaller; } table.footnote { font-size: calc(1em - 1pt); margin-block-start: 1em; position: relative } table.footnote::before { border: 1px solid black; content: ""; left: 0; position: absolute; top: 0; width: 25%; } td { /* pull text away from borders */ padding-inline: 0.5em; } td > p { /* remove top margin on a p inside table cell so that columns with paragraphs align at top of cell rather than leaving a blank margin */ margin-block-start: 0px; } input, textarea { border-width: 1px; } a.headerlink { font-size: 0.8em; margin-left: 0.3em; color: #c99; } div.admonition { padding-inline: 1em; padding-block: 0.25em; border-inline-start: grey solid 4px; background-color: #dfdfdf; } div.admonition.caution { border-inline-start: red solid 4px; } div.admonition.warning { border-inline-start: yellow solid 4px; } p.admonition-title { border-block-end: grey solid 2px; font-weight: bold; font-size: larger; margin-block: 0; } div.admonition.caution p.admonition-title { border-block-end: red solid 2px; } dt { font-weight: bold; margin-block-start: 1em;} dd p.first { margin-block-start: 0; } #skiplink { display: block; margin-block-start: 1em; margin-inline-start: 1em;} #skiplink a { /* force stuff for screenreader off screen */ position:absolute; left:-10000px; top:auto; width:1px; height:1px; overflow:hidden; padding: 0 0.75em; font-weight: bold; } #skiplink a:focus { position: static; width: auto; height: auto; } /* remove solid black border when focus is around the main section due to activation of skip-link. Page jumps there, that's enough. */ #main:focus-visible {outline: none;} /* Prevent examples from extending outside the viewport */ div.highlight > pre { overflow-wrap: anywhere; white-space: break-spaces; } code-1a221cf0758b/doc/_templates/layout.html0000644000000000000000000001601214427061170016750 0ustar 00000000000000{# _templates/layout.html ~~~~~~~~~~~~~~~~~~~~~~ Custom layout template for Roundup. #} {%- block doctype -%} {%- endblock %} {%- macro relbar(class) %}
    {%- for rellink in rellinks %}
  • {{ rellink[3] }} {%- if not loop.first %}{{ reldelim2 }}{% endif %}
    • {%- endfor %} {%- block rootrellink %}
  • {{ shorttitle|e }}{{ reldelim1 }}
    • {%- endblock %} {%- for parent in parents %}
  • {{ parent.title }}{{ reldelim1 }}
    • {%- endfor %} {%- block relbaritems %} {% endblock %}
{%- endmacro %} {%- macro sidebar() %} {%- block sidebartoc %} {%- if display_toc %}

{{ _('Table Of Contents') }}

{{ toc }} {%- endif %} {%- endblock %} {%- block sidebarrel %} {%- if prev %}

{{ _('Previous topic') }}

{{ prev.title }}

{%- endif %} {%- if next %}

{{ _('Next topic') }}

{{ next.title }}

{%- endif %} {%- endblock %} {%- block sidebarsourcelink %} {%- if show_source and has_source and sourcename %}

{{ _('This Page') }}

{%- endif %} {%- endblock %} {%- block sidebarsearch %} {%- if pagename != "search" %} {%- endif %} {%- endblock %} {%- endmacro %} {%- macro css() %} {%- for cssfile in css_files %} {%- endfor %} {%- endmacro %} {%- if 'name="description"' not in metatags %} {%- endif %} {{ metatags }} {%- if builder != 'htmlhelp' %} {%- set titlesuffix = " — "|safe + docstitle|e %} {%- endif %} {{ title|striptags }}{{ titlesuffix }} {%- if builder == 'web' %} {%- for link, type, title in page_links %} {%- endfor %} {%- else %} {{ css() }} {%- endif %} {%- if builder != 'htmlhelp' %} {%- for scriptfile in script_files %} {%- endfor %} {%- if use_opensearch %} {%- endif %} {%- if favicon %} {%- endif %} {%- endif %} {%- block linktags %} {%- if hasdoc('about') %} {%- endif %} {%- if hasdoc('copyright') %} {%- endif %} {%- if parents %} {%- endif %} {%- if next %} {%- endif %} {%- if prev %} {%- endif %} {%- endblock %} {%- block extrahead %} {% endblock %}
Roundup
{%- if pagename != "search" %} {%- endif %}
{{ sidebar() }}
{%- block content %}
{{ relbar('related-top') }} {% block body %} {% endblock %} {{ relbar('related-bottom') }}
{%- endblock %} {%- block footer %}
{%- if hasdoc('copyright') %} {% trans path=pathto('copyright'), copyright=copyright|e %}© Copyright {{ copyright }}.{% endtrans %} {%- else %} {% trans copyright=copyright|e %}© Copyright {{ copyright }}.{% endtrans %} {%- endif %}
{%- endblock %} code-1a221cf0758b/doc/acknowledgements.txt0000644000000000000000000002012214427061170016500 0ustar 00000000000000.. meta:: :description: Display contributions of code and documentation to the Roundup Issue Tracker. Includes users who reported issues. Updated for each release. Acknowledgements ================ The Roundup Initative is a group of people that regularly work together to produce new releases of the Roundup Issue Tracker. We like to thank our community and all organisation and people that support us doing so with code, money, time, testing, reports, ideas and everything else that helped! `Announcement with changelog for current release.`_ .. _`Announcement with changelog for current release.`: announcement.html 2.2 --- 2.2.0 ~~~~~ Maintainer: John Rouillard, Ralf Schlatterbeck Release Manager: John Rouillard Developer activity by changesets:: rouilj@ieee.org 291 ************************************************ rsc@runtux.com 11 ** aburke 1 cmeerw@cmeerw.org 1 thomas@intevation.de 1 Other contributers Marcus Priesch, John Kristensen (jerrykan), Ashely Burke, Nagy Gabor, Norbert Schlemmer, Thomas Arendsen Hein 2.1 --- 2.1.0 ~~~~~ Maintainer: John Rouillard, Ralf Schlatterbeck Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 178 *************************************************** rsc at runtux.com 29 ******** cedric.krier at b2ck.com 6 ** cmeerw at cmeerw.org 1 john at jerrykan.com 1 Tobias Herp 1 2.0 --- 2.0.0 ~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 62 *************************************************** rsc at runtux.com 11 ********* cmeerw at cmeerw.org 4 *** 2.0.0b0 ~~~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 133 ************************************************ cmeerw at cmeerw.org 35 ************* rsc at runtux.com 10 **** jsm at polyomino.org.uk 1 2.0.0a0 ~~~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj@ieee.org 13674 *************************************************************************************** jsm@polyomino.org.uk 4746 ****************************** cmeerw@cmeerw.org 3622 *********************** dangchau1991@yahoo.com 3374 ********************* rsc@runtux.com 2871 ****************** ezio.melotti@gmail.com 103 * john@jerrykan.com 40 bernhard@intevation.de 29 martin.v.loewis 8 cedric.krier@b2ck.com 6 tekberg@uw.edu 6 gabriele.roeger@unibas.ch 4 Other contributers Chau Nguyen, Andreas (anrounham14), Robert Klonner, Garth Jensen 1.6 --- 1.6.1 ~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 19 ********************************************* bernhard at intevation.de 1 ** jsm at polyomino.org.uk 1 ** Joseph Myers, Cédric Krier, hannob/JulienPalard 1.6.0 ~~~~~ Maintainer: Ralf Schlatterbeck, John Rouillard Release Manager: John Rouillard Developer activity by changesets:: rouilj at ieee.org 234 ********************************************** rsc at runtux.com 30 ****** john at jerrykan.com 26 ***** bernhard at intevation.de 10 ** esr at thyrsus.com 8 ** techtonik at gmail.com 5 * dangchau1991 at gmail.com 1 pf at artcom-gmbh.de 1 thomas at intevation.de 1 Frank Niessink, Anthony (antmail), Bruce Tulloch, kinggreedy, David Wolever, Tom Ekberg, Peter Funk, Daniel Diniz, W. Trevor King, Igor Ippolitov, Joseph Myers, r.david.murray, martin.v.loewis, Henry (henryl), David Wolever, Stuart McGraw (smcgraw), Philipp Gortan (mephinet), Ezio Melotti, Heiko Stegmann, Trent Gamblin, Anton Schur 1.5 --- 1.5.1 ~~~~~ Maintainer: Ralf Schlatterbeck Release Manager: Anatoly Techtonik Developer activity by changesets:: hg churn -c --rev 1.5.0:1.5.1 techtonik@gmail.com 98 ********************************************** bernhard@intevation.de 44 ********************* Bernhard Reiter rsc@runtux.com 37 ***************** Ralf Schlatterbeck john@jerrykan.com 26 ************ John Kristensen thomas@intevation.de 22 ********** Thomas Arendsen Hein rouilj@ieee.org 5 ** John P. Rouillard caulagi@gmail.com 2 * Pradip Caulagi ced@b2ck.com 1 Cédric Krier Mathias Behrle, Ezio Melotti, R David Murray, James Mack, Thibault Fevry, Kai Storbeck, Kay Hayen, Karl-Philipp Richter 1.5.0 ~~~~~ Maintainer: Ralf Schlatterbeck:: hg churn -c --rev 1.4.21:1.5.0 techtonik@gmail.com 37 ********************************************** Anatoly Techtonik bernhard@intevation.de 17 ********************* Bernhard Reiter caulagi@gmail.com 16 ******************** Pradip Caulagi rouilj@ieee.org 8 ********** John P. Rouillard rsc@runtux.com 6 ******* Ralf Schlatterbeck thomas@intevation.de 5 ****** Thomas Arendsen Hein Tom Ekberg, Erik Hanspers pre 1.5.0 --------- Go Ping, you rock! Also, go Common Ground, ekit.com and Bizar Software for letting me implement this system on their time. Thanks also to the many people on the mailing list, in the sourceforge project and those who just report bugs: Christian Aastorp, Thomas Arendsen Hein, Nerijus Baliunas, Benni Bärmann, Anthony Baxter, David Benjamin, Marlon van den Berg, Bo Berglund, Olly Betts, Stéphane Bidoul, Rafal Bisingier, Cameron Blackwood, Jeff Blaine, Duncan Booth, Seb Brezel, J Alan Brogan, Titus Brown, Steve Byan, Brett Cannon, Godefroid Chapelle, Pradip Caulagi, Eli Collins, Roch'e Compaan, Wil Cooley, Joe Cooper, Kelley Dagley, Bruno Damour, Bradley Dean, Toby Dickenson, Paul F. Dubois, Hauke Duden, Eric Earnst, Peter Eisentraut, Andrew Eland, Jeff Epler, Tom Epperly, Tamer Fahmy, Thibault Févry, Vickenty Fesunov, Hernan Martinez Foffani, Trent Gamblin, Stuart D. Gathman, Martin Geisler, Ajit George, Dirk Geschke, Frank Gibbons, Johannes Gijsbers, Christian Glass, Gus Gollings, Philipp Gortan, Dan Grassi, Robin Green, Jason Grout, Charles Groves, Engelbert Gruber, Bruce Guenter, Tamás Gulácsi, Satchidanand Haridas, Sebastian Harl, Ralf Hemmecke, Juergen Hermann, Tobias Herp, Uwe Hoffmann, Alex Holkner, Tobias Hunger, Werner Hunger, Simon Hyde, Paul Jimenez, Christophe Kalt, Timo Kankare, Brian Kelley, James Kew, Sheila King, Michael Klatt, Bastian Kleineidam, Robert Klonner, Axel Kollmorgen, Cédric Krier, John Kristensen, Detlef Lannert, Andrey Lebedev, Henrik Levkowetz, David Linke, Martin v. Löwis, Fredrik Lundh, Jochen Maes, Will Maier, Ksenia Marasanova, Georges Martin, Gordon McMillan, Christof Meerwald, John F Meinel Jr, Roland Meister, Ezio Melotti, Ulrik Mikaelsson, John Mitchell, Ramiro Morales, Toni Mueller, Joseph Myers, Chau Nguyen, Stefan Niederhauser, Truls E. Næss, Bryce L Nordgren, Patrick Ohly, "om", Luke Opperman, Anthony Pankov, Eddie Parker, Will Partain, Timo Paulssen, Benjamin Pollack, Peter Pöml, Ewout Prangsma, Marcus Priesch, Eric S. Raymond, Bernhard Reiter, Roy Rapoport, John P. Rouillard, Luke Ross, Jesse Ruderman, Ollie Rutherfurd, Toby Sargeant, Giuseppe Scelsi, Ralf Schlatterbeck, Gregor Schmid, Florian Schulze, Klamer Schutte, Dougal Scott, Stefan Seefeld, Jouni K Seppänen, Jeffrey P Shell, Dan Shidlovsky, Joel Shprentz, Terrel Shumway, Emil Sit, Alexander Smishlajev, Nathaniel Smith, Leonardo Soto, Maciej Starzyk, Heiko Stegmann, Kai Storbeck, Mitchell Surface, anatoly techtonik, Jon C. Thomason, Mike Thompson, Hubert Touvet, Michael Twomey, Joseph E. Trent, Karl Ulbrich, Martin Uzak, Darryl VanDorp, J Vickroy, Timothy J. Warren, Jakub Wilk, William (Wilk), Tue Wennerberg, Matt Wilbert, Chris Withers, David Wolever, Cheer Xiao, Milan Zamazal. code-1a221cf0758b/doc/admin_guide.txt0000644000000000000000000014604214427061170015425 0ustar 00000000000000.. meta:: :description: How to administer, backup, upgrade a Roundup installation. System and user security, configuring web compression, documentation on using roundup-server and running roundup-admin. ==================== Administration Guide ==================== .. contents:: :local: What does Roundup install? ========================== There's two "installations" that we talk about when using Roundup: 1. The installation of the software and its support files. This uses the standard Python mechanism called "setuptools" and thus Roundup's core code, executable scripts and support data files are installed in Python's directories. On Windows, this is typically: Scripts ``\scripts\...`` Core code ``\lib\site-packages\roundup\...`` Support files ``\share\roundup\...`` and on Unix-like systems (eg. Linux): Scripts ``/bin/...`` Core code ``/lib-/site-packages/roundup/...`` Support files ``/share/roundup/...`` 2. The installation of a specific tracker. When invoking the roundup-admin "inst" (and "init") commands, you're creating a new Roundup tracker. This installs configuration files, HTML templates, detector code and a new database. You have complete control over where this stuff goes through both choosing your "tracker home" and the ``main`` -> ``database`` variable in the tracker's config.ini. Configuring Roundup's Logging of Messages For Sysadmins ======================================================= You may configure where Roundup logs messages in your tracker's config.ini file. Roundup will use the standard Python (2.3+) logging implementation. Configuration for standard "logging" module: - tracker configuration file specifies the location of a logging configration file as ``logging`` -> ``config`` - ``roundup-server`` specifies the location of a logging configuration file on the command line Configuration for "BasicLogging" implementation: - tracker configuration file specifies the location of a log file ``logging`` -> ``filename`` - tracker configuration file specifies the level to log to as ``logging`` -> ``level`` - ``roundup-server`` specifies the location of a log file on the command line - ``roundup-server`` specifies the level to log to on the command line (``roundup-mailgw`` always logs to the tracker's log file) In both cases, if no logfile is specified then logging will simply be sent to sys.stderr with only logging of ERROR messages. Configuring roundup-server ========================== The basic configuration file is as follows (taken from the ``roundup-server.ini.example`` file in the "doc" directory):: [main] # Host name of the Roundup web server instance. # If left unconfigured (no 'host' setting) the default # will be used. # If empty, listen on all network interfaces. # If you want to explicitly listen on all # network interfaces, the address 0.0.0.0 is a more # explicit way to achieve this, the use of an empty # string for this purpose is deprecated and will go away # in a future release. # Default: localhost host = localhost # Port to listen on. # Default: 8080 port = 8017 # Path to favicon.ico image file. If unset, built-in favicon.ico is used. # The path may be either absolute or relative # to the directory containing this config file. # Default: favicon.ico favicon = favicon.ico # User ID as which the server will answer requests. # In order to use this option, the server must be run initially as root. # Availability: Unix. # Default: user = roundup # Group ID as which the server will answer requests. # In order to use this option, the server must be run initially as root. # Availability: Unix. # Default: group = # don't fork (this overrides the pidfile mechanism)' # Allowed values: yes, no # Default: no nodaemon = no # Log client machine names instead of IP addresses (much slower) # Allowed values: yes, no # Default: no log_hostnames = no # File to which the server records the process id of the daemon. # If this option is not set, the server will run in foreground # # The path may be either absolute or relative # to the directory containing this config file. # Default: pidfile = # Log file path. If unset, log to stderr. # The path may be either absolute or relative # to the directory containing this config file. # Default: logfile = # Set processing of each request in separate subprocess. # Allowed values: debug, none, thread, fork. # Default: fork multiprocess = fork # Tracker index template. If unset, built-in will be used. # The path may be either absolute or relative # to the directory containing this config file. # Default: template = # Enable SSL support (requires pyopenssl) # Allowed values: yes, no # Default: no ssl = no # PEM file used for SSL. A temporary self-signed certificate # will be used if left blank. # The path may be either absolute or relative # to the directory containing this config file. # Default: pem = # Roundup trackers to serve. # Each option in this section defines single Roundup tracker. # Option name identifies the tracker and will appear in the URL. # Option value is tracker home directory path. # The path may be either absolute or relative # to the directory containing this config file. [trackers] demo = /trackers/demo sysadmin = /trackers/sysadmin Additional notes for each keyword: **template** Specifies a template used for displaying the tracker index when multiple trackers are being used. It is processed by TAL and the variable "trackers" is available to the template and is a dict of all configured trackers. **ssl** Enables use of SSL to secure the connection to the roundup-server. In most cases, you will want to run a real web server (Apache, Nginx) as a proxy to roundup-server running without SSL. The real web server can filter/rate limit/firewall requests to roundup-server. If you enable this, ensure that your tracker's config.ini specifies an *https* URL. See roundup-server.1 man page for additional information. **pem** If specified, the SSL PEM file containing the private key and certificate. The file must include both the private key and certificate with appropriate headers (e.g. ``-----BEGIN PRIVATE KEY-----``, ``-----END PRIVATE KEY-----`` and ``-----BEGIN CERTIFICATE-----``, ``-----END CERTIFICATE-----``. If not specified, roundup will generate a temporary, self-signed certificate for use. **trackers** section Each line denotes a mapping from a URL component to a tracker home. Make sure the name part doesn't include any url-unsafe characters like spaces. Stick to alphanumeric characters and you'll be ok. To generate a config.ini in the current directory (note it will overwrite an existing file) from the roundup-server command line use:: roundup_server -p 8017 -u roundup --save-config demo=/trackers/demo \ sysadmin=/trackers/sysadmin Configuring Compression ======================= Roundup will compress HTTP responses to clients on the fly. Dynamic, on the fly, compression is enabled by default, to disable it set:: [web] ... dynamic_compression = No in the tracker's ``config.ini``. You should disable compression if your proxy (e.g. nginx or apache) or wsgi server (uwsgi) is configured to compress responses on the fly. The python standard library includes gzip support. For brotli or zstd you will need to install packges. See the `installation documentation`_ for details. Some assets will not be compressed on the fly. Assets with mime types of "image/png" or "image/jpeg" will not be compressed. You can add mime types to the list by using ``interfaces.py`` as discussed in the `customisation documentation`_. As an example adding:: from roundup.cgi.client import Client Client.precompressed_mime_types.append('application/zip`) to ``interfaces.py`` will prevent zip files from being compressed. Any content less than 100 bytes in size will not be compressed (e.g errors messages, short json responses). Zstd will be used if the client can understand it, followed by brotli then gzip encoding. Currently the preference order is hard coded into the server and not parsed using ``q`` values from the client's Accept-Encoding header. This is an area for improvement. In addition to dynamic compression, static files/assets accessed using ``@@file`` can be pre-compressed. This reduces CPU load on the server and reduces the time required to respond to the client. By default searching for pre-compressed files is disabled. To enable it set:: [web] ... use_precompressed_files = Yes in the tracker's ``config.ini`` file. Then you can create a precompressed file and it will be served if the client is able to accept it. For a file ``.../@@file/library.js`` you can create:: tracker_home/html/library.js.gzip tracker_home/html/library.js.br tracker_home/html/library.js.zstd which should be created by using (respectively):: gzip --keep --suffix .gzip library.js brotli library.js zstd library.js && mv library.js.zst library.js.zstd see the man pages for options that control compression level. Note that some levels require additional memory on the client side, so you may not always want to use the highest compression available. A pre-compressed file will not be used if its modified date is earlier than the uncompressed file. For example, if ``library.js.gzip`` is older (has earlier modification date) than ``library.js``, ``library.js.gzip`` will be ignored. ``library.js`` will be served instead. ``library.js`` will be dynamically compressed on the fly and a warning message will be logged. Precompressed files override dynamic compression. For example, assume the client can accept brotli and gzip. If there are no precompressed files, the data will be compressed dynamically (on the fly) using brotli. If there is a precompressed gzip file present the client will get the gzip version and not a brotli compressed version. This mechanism allows the admin to allow use of brotli and zstd for dynamic content, but not for static content. Adding a Web Content Security Policy (CSP) ========================================== A Content Security Policy (`CSP`_) adds a layer of security to Roundup's web interface. It makes it more difficult for an attacker to compromise Roundup. By default Roundup does not add a CSP. If you need to implement a CSP, this section will help you understand how to add one and document the current level of support for CSP in Roundup. Roundup's web interface has remained mostly unchanged since it was created over a decade ago. Current releases have been slowly modernizing the HTML to improve security. There are still some improvements that need to happen before the tightest CSP configurations can be used. Writing a CSP is complex. This section just touches on how to create and install a CSP to improve security. Some of it might break functionality. There are two ways to add a CSP: 1. a fixed CSP added by a server 2. a dynamic CSP added by Roundup Fixed CSP --------- If you are using a web server (Apache, Nginx) to run Roundup, you can add a ``Content-Security-Policy`` header using that server. WSGI servers like uWSGI can also be configured to add headers. An example header would look like:: Content-Security-Policy: default-src 'self' 'unsafe-inline' 'strict-dynamic'; One thing that may need to be included is the ``unsafe-inline``. The default templates use ``onload``, ``onchange``, ``onsubmit``, and ``onclick`` JavaScript handlers. Without ``unsafe-inline`` these won't work and popup helpers will not work. Sadly the use of ``unsafe-inline`` is a pretty big hole in this CSP. You can set the hashes for all the JavaScript handlers in the CSP. Then replace ``unsafe-inline`` with ``unsafe-hashes`` to help close this hole, but has its own issues. See `remediating unsafe-inline`_ for another way to mitigate this. The inclusion of ``strict-dynamic`` allows trusted JavaScript files that are downloaded from Roundup to make changes to the web interface. These changes are also trusted code that will be run when invoked. More secure CSPs can also be created. However because of the ability to customize the web interface, it is difficult to provide guidance. Dynamic CSP ----------- Roundup creates a cryptographic nonce for every client request. The nonce is the value of the ``client.client_nonce`` property. By changing the templates to use the nonce, we can better secure the Roundup instance. However the nonce has to be set in the CSP returned by Roundup. One way to do this is to add a templating utility to the extensions directory that generates the CSP on the fly. For example:: default_security_headers = { 'Content-Security-Policy': ( "default-src 'self'; " "base-uri 'self'; " "script-src https: 'nonce-{nonce}' 'strict-dynamic'; " "style-src 'self' 'nonce-{nonce}'; " "img-src 'self' data:; " "frame-ancestors 'self'; " "object-src 'self' 'nonce-{nonce}'; " ), } def AddHtmlHeaders(client, header_dict=None): ''' Generate https headers from dict use default security headers Setting the header with a value of None will not inject the header and can override the default set. Header values will be formatted with a dictionary including a nonce. Use to set a nonce for inline scripts. ''' try: if client.client_nonce is None: # logger.warning("client_nonce is None") client.client_nonce = client.session_api._gen_sid() except AttributeError: # client.client_nonce doesn't exist, create it # logger.warning("client_nonce does not exist, creating") client.client_nonce = client.session_api._gen_sid() headers = default_security_headers.copy() if isinstance(header_dict, dict): headers.update(header_dict) client_headers = client.additional_headers for header, value in list(headers.items()): if value is None: continue client_headers[header] = value.format( nonce=client.client_nonce) def init(instance): instance.registerUtil('AddHtmlHeaders', AddHtmlHeaders) Adding the following to ``page.html`` right after the opening ```` tag:: will invoke ``AddHtmlHeaders()`` to add the CSP header with the nonce. With this set of CSP headers, all style, script and object tags will need a ``nonce`` attribute. This can be added by changing:: to:: for each script, object or style tag. Remediating ``unsafe-inline`` ----------------------------- .. _remediating unsafe-inline: Using a trusted script to set event handlers to replace the ``onX`` handlers allows removal of the ``unsafe-inline`` handlers. If you remove ``unsafe-inline`` the ``onX`` handlers will not run. However you can use the label provided by the ``onX`` attribute to securely enable a callback function. This method is a work in progress. As an example proof of concept, adding this "decorator" script at the end of page.html:: will set callback for the submit even on any form that has an onsubmit attribute to ``submit_once()``. ``submit_once`` is defined in Roundup's base_javascript and is generated with a proper nonce. By including the nonce in the dynamic CSP, we can use our trusted "decorator" script to add event listeners. These listeners will call the trusted function in base_javascript to replace the ignored ``onX`` handlers. .. _CSP: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP Configuring native-fts Full Text Search ======================================= Roundup release 2.2.0 supports database-native full text search. SQLite (minimum version 3.9.0) with FTS5 and PostgreSQL (minimum version 11.0) with websearch_to_tsvector are supported. To enable this method, change the ``indexer`` setting in the tracker's config.ini to ``native-fts``. Then reindex using ``roundup-admin -i tracker_home reindex``. The amount of time it takes to reindex depends on the amount of data in your tracker, the speed of your disks, etc. It can take hours. SQLite details -------------- The SQLite native-fts changes the full text search query a little bit. For the other search methods, the search terms are split on white space and each item in the index: a field (e.g. title), message content and file content is searched for all the terms. If any term is missing that item is ignored. Once the items are found they are mapped to an issue and the normal issue index is displayed. When using FTS5, the search terms can use the full text search query language described at: https://www.sqlite.org/fts5.html#full_text_query_syntax. This supports: * plain word search (joined with and similar to other search methods) * phrase search with terms enclosed in quotes (``"``) * proximity search with varying distances using ``NEAR()`` * boolean operations by grouping with parentheses and using ``AND`` and ``OR`` * exclusion using ``NOT`` * prefix searching by prefixing the term with``^`` All of the data that is indexed is in a single column, so when column specifiers are used they usually result in an error which is detected and an enhanced error message is produced. Unlike the native, xapian and whoosh indexers there is no limit to the length of terms that are indexed. Also stopwords are indexed but ignored when searching if they are the only word in the search. So a search for "the" will return no results but "the book" will return results. Pre-filtering the stopwords when indexing would break proximity and phrase searching. This may be helpful or problematic for your particular tracker. To support the most languages available, the unicode61 tokenizer is used without porter stemming. Using the ``indexer_language`` setting to enable stemming for ``english`` is not available in this implementation. Also ranking information is not used in this implementation. These are areas for improvement. PostgreSQL info --------------- The PostgreSQL native-fts changes the full text search query a little bit. When using PostgreSQL full text search, two different query languages are supported. 1. websearch - described at the end of `Parsing Queries`_ under websearch_to_tsquery. This is the default. 2. tsquery - described at the beginning of `Parsing Queries`_ with to_tsquery. It is enabled by starting the search phrase with ``ts:``. .. _Parsing Queries: \ https://www.postgresql.org/docs/14/textsearch-controls.html#TEXTSEARCH-PARSING-QUERIES Websearch provides a more natural style of search and supports: * plain word search (stemmed in most cases) * phrase search with terms enclosed in quotes (``"``) * exclusion by prefixing a term/phrase with ``-`` * alternative/or searching with ``or`` between terms * ignores non-word characters including punctuation Tsquery supports: * a strict query syntax * plain word search * phrase search with the ``<->`` operator or enclosing the phrase in ``'`` single quotes (note this will use a stemmer on the terms in the phrase). * proximity search with varying distances using ```` * boolean operations by grouping with parentheses and using ``&`` and ``|`` * exclusion using ``!`` * prefix searching using ``:*`` at the end of the prefix All of the data that is indexed is in a single column and input weighing is not used. Depending on the FTS configuration (determined by the ``indexer_language`` setting), stopwords are supported. PostgreSQL takes the stopwords into account when calculating the data needed for proximity/near searches. Like SQLite FTS, there is no limit to the length of terms that are indexed. Again this may be helpful or problematic for your particular tracker. The config.ini ``indexer_language`` setting is used to define the configuration used for indexing. For example with the default ``english`` setting a snowball stemmer (english_stem) is used. So words like 'drive' and 'driving' and 'drive-in' will all match a search for 'drive' but will not match 'driver'. The indexer_language is used as the configuration name for every call to the text search functions (to_tsvector, to_tsquery). Changing this requires reindexing. The `configuration list can be obtained using using psql's`_ ``\dF`` command. .. _configuration list can be obtained using using psql's: \ https://www.postgresql.org/docs/current/textsearch-psql.html Roundup includes a hardcoded list for all languages supported by PostgreSQL 14.1. The list includes 5 custom "languages" ``custom1`` ... ``custom5`` to allow you to set up your `own textsearch configuration`_ using one of the custom names. Depending on your PostgreSQL version, we may allow an invalid language to be configured. You will see an error about ``text search configuration ... does not exist``. .. _own textsearch configuration: \ https://www.postgresql.org/docs/14/textsearch-configuration.html It may be possible to append to this list using the tracker's interfaces.py. For details, see ``test/test_indexer.py`` in the roundup distribution and search for ``valid_langs``. If you succeed please email roundup-users AT lists.sourceforge.net with a description of your success. After changing the configuration language, you must reindex the tracker since the index must match the configuration language used for querying. Also there are various `dictionaries`_ that allow you to: * add stopwords * override stemming for a term * add synonyms (e.g. a search for "pg" can also match 'psql' "postgresql") * add terms that expand/contract the search space (Thesaurus dictionary) * additional transforms .. _dictionaries: https://www.postgresql.org/docs/14/textsearch-dictionaries.html Use of these is beyond this documentation. Please visit the appropriate PostgreSQL documents. The following my also be helpful: * https://rachbelaid.com/postgres-full-text-search-is-good-enough/ Ranking information is not used in this implementation. Also stop words set in config.ini are ignored. These are areas for improvement. Cleaning up old native indexes ------------------------------ If you are happy with the database fts indexing, you can save some space by removing the data from the native text indexing tables. This requires using the ``sqlite3`` or ``psql`` commands to execute SQL to delete the rows in the ``__textids`` and ``__words`` tables. You can do this with the following SQL commands:: delete from __words; delete from __textids; Note this deletes data from the tables and does *not* delete the table. This allows you to revert to Roundup's native full text indexing on SQLite or Postgres. If you were to delete the tables, Roundup will not recreate the tables. Under PostgreSQL, you can use the ``truncate `` command if you wish. Configuring Session Databases ============================= The session and OTK (one time key) databases store information about the operation of Roundup. This ephemeral data: * web login session keys, * CSRF tokens, * email password recovery one time keys, * rate limiting data, * ... can be a performance bottleneck. It usually happens with anydbm or SQLite backends. PostgreSQL and MySQL are sufficiently powerful that they can handle the higher transaction rates. If you are using sqlite, you can choose to use the anydbm database for session data. By default it will use additional sqlite databases for storing the session and otk data. The following table shows which primary databases support different session database backends:: main\/ session>| anydbm | sqlite | redis | mysql | postgresql | anydbm | D | | X | | | sqlite | X | D | X | | | mysql | | | | D | | postgresql | | | | | D | --------------------------------------------------------------+ D - default if unconfigured, X - compatible choice The ``backend`` setting is in the tracker's ``config.ini`` file under the ``sessiondb`` section. Using Redis for Session Databases --------------------------------- Redis is an in memory key/value data structure store. You need to install the redis-py_ module from pypi. Then install Redis using your package manager or by downloading it from the Redis_ website. You need to secure your redis instance. The data that Roundup stores includes session cookies and other authentication tokens. At minimum you should require a password to connect to your redis database. Set ``requirepass`` in ``redis.conf``. Then change the ``redis_url`` in ``config.ini`` to use the password. For example:: redis://:mypassword@localhost:7200/10 will connect to the redis instance running on localhost at port 7200 using the password ``mypassword`` to open database 10. The ``redis_url`` setting can load a file to better secure the url. If you are using redis 6.0 or newer, you can specify a username/password and access control lists to improve the security of your data. Another good alternative is to talk to redis using a Unix domain socket. If you are connecting to redis across the network rather than on localhost, you should configure ssl/tls and use the ``rediss`` scheme in the url along with the query parameters:: ssl_cert_reqs=required&ssl_ca_certs=/path/to/custom/ca-cert where you specify the file that can be used to validate the SSL certificate. `Securing Redis`_ has more details. .. _Redis: https://redis.io .. _redis-py: https://pypi.org/project/redis/ .. _Securing Redis: https://redis.io/docs/manual/security/ Users and Security ================== Roundup holds its own user database which primarily contains a username, password and email address for the user. Roundup *must* have its own user listing, in order to maintain internal consistency of its data. It is a relatively simple exercise to update this listing on a regular basis, or on demand, so that it matches an external listing (eg. :ref:`unix passwd file`, `LDAP `_, etc.) Roundup identifies users in a number of ways: 1. Through the web, users may be identified by either HTTP Basic Authentication or cookie authentication. If you are running the web server (roundup-server) through another HTTP server (eg. apache or IIS) then that server may require HTTP Basic Authentication, and it will pass the ``REMOTE_USER`` variable (or variable defined using http_auth_header) through to Roundup. If this variable is not present, then Roundup defaults to using its own cookie-based login mechanism. 2. In email messages handled by roundup-mailgw, users are identified by the From address in the message. In both cases, Roundup's behaviour when dealing with unknown users is controlled by Permissions defined in the "SECURITY SETTINGS" section of the tracker's ``schema.py`` module: Web Access and Register If granted to the Anonymous Role, then anonymous users will be able to register through the web. Email Access and Register If granted to the Anonymous Role, then email messages from unknown users will result in those users being registered with the tracker. More information about how to customise your tracker's security settings may be found in the `customisation documentation`_. Configuring Authentication Header/Variable ------------------------------------------ The front end server running Roundup can perform the user authentication. It pass the authenticated username to the backend in a variable. By default roundup looks for the ``REMOTE_USER`` variable This can be changed by setting the parameter ``http_auth_header`` in the ``[web]`` section of the tracker's ``config.ini`` file. If the value is unset (the default) the REMOTE_USER variable is used. If you are running roundup using ``roundup-server`` behind a proxy that authenticates the user you need to configure ``roundup-server`` to pass the proper header to the tracker. By default ``roundup-server`` looks for the ``REMOTE_USER`` header for the authenticated user. You can copy an arbitrary header variable to the tracker using the ``-I`` option to roundup-server (or the equivalent option in the roundup-server config file). For example to use the ``uid_variable`` header, two configuration changes are needed: First configure ``roundup-server`` to pass the header to the tracker using:: roundup-server -I uid_variable .... note that the header is passed exactly as supplied by the upstream server. It is **not** prefixed with ``HTTP_`` like other headers since you are explicitly whitelisting the header. Multiple comma separated headers can be passed to the ``-I`` option. These could be used in a detector or other tracker extensions, but only one header can be used by the tracker as an authentication header. To make the tracker honor the new variable changing the tracker ``config.ini`` to read:: [web] ... http_auth_header = uid_variable At the time this is written, support is experimental. If you use it you should notify the roundup maintainers using the roundup-users mailing list. Securing Secrets ================ Roundup can read secrets from a file that is referenced from any of the config.ini files. If you use Docker, you can bind mount the files from a secure location, or store them in a subdirectory of the tracker home. You can also use a secrets management tool like Docker Swarm's secrets management. This example config.ini configuration gets the database password from a file populated by Swarm secrets:: [rdbms] # Database user password. # A string that starts with 'file://' is interpreted as a file # path relative to the tracker home. Using 'file:///' defines # an absolute path. The first line of the file will be used as # the value. Any string that does not start with 'file://' is # used as is. It removes any whitespace at the end of the # line, so a newline can be put in the file. # # Default: roundup password = file:///run/secrets/db_password assuming that Docker Swarm secrets has the key ``db_password`` and the ``--secret db_password`` option is used when starting the Roundup service. Because environment variables can be inadvertently exposed in logs or process listings, Roundup does not currently support loading secrets from environment variables. Tasks ===== Maintenance of Roundup can involve one of the following: 1. `tracker backup`_ 2. `software upgrade`_ 3. `migrating backends`_ 4. `moving a tracker`_ 5. `migrating from other software`_ 6. `adding a user from the command-line`_ Tracker Backup -------------- The roundup-admin import and export commands are **not** recommended for performing backup. Optionally stop the web and email frontends and to copy the contents of the tracker home directory to some other place using standard backup tools. This means using *pg_dump* to take a snapshot of your Postgres backend database, for example. A simple copy of the tracker home (and files storage area if you've configured it to be elsewhere) will then complete the backup. Software Upgrade ---------------- .. _make a backup: #tracker-backup Always `make a backup`_ of your tracker before upgrading software. Steps you may take: 1. Install pytest and ensure that the unit tests run on your system (using your preferred python version):: pip2 install pytest python2 -m pytest test/ pip3 install pytest python3 -m pytest test/ 2. If you're using an RDBMS backend, make a backup of its contents now. 3. Make a backup of the tracker home itself. 4. Stop the tracker web and email frontends. 5. Install the new version of the software:: python setup.py install 6. Follow the steps in the `upgrading documentation`_ for all the versions between your original version and the new version. Usually you should run `roundup_admin -i migrate` on your tracker(s) before you allow users to start accessing the tracker. It's safe to run this even if it's not required, so just get into the habit. 7. Restart your tracker web and email frontends. If something bad happens, you may reinstate your backup of the tracker and reinstall the older version of the sofware using the same install command:: python setup.py install .. index:: database; convert from one database backend to another single: roundup-admin; import and export Migrating Backends ------------------ 1. Stop the existing tracker web and email frontends (preventing changes). 2. Use the roundup-admin tool "export" command to export the contents of your tracker to disk. (If you are running on windows see `issue1441336 `_ on how to use the command line rather than interactive mode to export data.) 3. Copy the tracker home to a new directory. 4. Delete the "db" directory from the new directory. 5. Set the value of the ``backend`` key under the ``[database]`` section of the tracker's ``config.ini`` file. 6. Use the roundup-admin "import" command to import the previous export with the new tracker home. If non-interactively:: roundup-admin -i import If interactively, enter 'commit' before exiting. 7. Test each of the admin tool, web interface and mail gateway using the new backend. 8. Move the old tracker home out of the way (rename to "tracker.old") and move the new tracker home into its place. 9. Restart web and email frontends. Moving a Tracker ---------------- If you're moving the tracker to a similar machine, you should: 1. install Roundup on the new machine and test that it works there, 2. stop the existing tracker web and email frontends (preventing changes), 3. copy the tracker home directory over to the new machine, and 4. start the tracker web and email frontends on the new machine. Most of the backends are actually portable across platforms (ie. from Unix to Windows to Mac). If this isn't the case (ie. the tracker doesn't work when moved using the above steps) then you'll need to: 1. install Roundup on the new machine and test that it works there, 2. stop the existing tracker web and email frontends (preventing changes), 3. use the roundup-admin tool "export" command to export the contents of the existing tracker, 4. copy the export to the new machine, 5. use the roundup-admin "import" command to import the tracker on the new machine, and 6. start the tracker web and email frontends on the new machine. .. index:: pair: roundup; migrate from other bugtracker software Migrating From Other Software ----------------------------- You have a couple of choices. You can either use a CSV import into Roundup, or you can write a simple Python script which uses the Roundup API directly. The latter is almost always simpler -- see the "scripts" directory in the Roundup source for some example uses of the API. "roundup-admin import" will import data into your tracker from a directory containing files with the following format: - one colon-separated-values file per Class with columns for each property, named .csv - one colon-separated-values file per Class with journal information, named -journals.csv (this is required, even if it's empty) - if the Class is a FileClass, you may have the "content" property stored in separate files from the csv files. This goes in a directory structure:: -files// where ```` is the item's ```` combination. The ```` value is ``int( / 1000)``. .. index:: pair: roundup-admin; managing users Adding A User From The Command-Line ----------------------------------- The ``roundup-admin`` program can create any data you wish to in the database. To create a new user, use:: roundup-admin create user To figure out what good values might be for some of the fields (eg. Roles) you can just display another user:: roundup-admin list user (or if you know their username, and it happens to be "richard"):: roundup-admin filter user username=richard then using the user id (e.g. 5) you get from one of the above commands, you may display the user's details:: roundup-admin display where designator is ``user5``. Running the Servers =================== Unix ---- On Unix systems, use the scripts/server-ctl script to control the roundup-server server. Copy it somewhere and edit the variables at the top to reflect your specific installation. If you use systemd look at scripts/systemd.gunicorn. It is configured for a wsgi deployment using gunicorn, but may be a good starting point for your setup. Windows ------- On Windows, the roundup-server program runs as a Windows Service, and therefore may be controlled through the Services control panel. Note that you **must** install the pywin32 package to allow roundup to run as a service. The roundup-server program may also control the service directly: **install the service** ``roundup-server -C /path/to/my/roundup-server.ini -c install`` **start the service** ``roundup-server -c start`` **stop the service** ``roundup-server -c stop`` To bring up the services panel: Windows 2000 and later Start/Control Panel/Administrative Tools/Services Windows NT4 Start/Control Panel/Services You will need a server configuration file (as described in `Configuring roundup-server`_) for specifying tracker homes and other roundup-server configuration. Specify the name of this file using the ``-C`` switch when installing the service. Running the Mail Gateway Script ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The mail gateway script should be scheduled to run regularly on your Windows server. Normally this will result in a window popping up. The solution to this is to: 1. Create a new local account on the Roundup server 2. Set the scheduled task to run in the context of this user instead of your normal login Mail gateway script command line ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Usage:: usage: roundup_mailgw.py [-h] [-v] [-c DEFAULT_CLASS] [-I OAUTH_CLIENT_ID] [-O OAUTH_DIRECTORY] [-S SET_VALUE] [-T OAUTH_TOKEN_ENDPOINT] [args ...] The roundup mail gateway may be called in one of three ways: - without arguments. Then the env var ROUNDUP_INSTANCE will be tried. - with an instance home as the only argument, - with both an instance home and a mail spool file, or - with an instance home, a mail source type and its specification. It also supports optional ``-S`` (or ``--set-value``) arguments that allows you to set fields for a class created by the roundup-mailgw. The format for this option is [class.]property=value where class can be omitted and defaults to msg. The ``-S`` options uses the same property=value[;property=value] notation accepted by the command line roundup command or the commands that can be given on the Subject line of an email message (if you're using multiple properties delimited with a semicolon the class must be specified only once in the beginning). It can let you set the type of the message on a per e-mail address basis by calling roundup-mailgw with different email addresses and other settings. PIPE: If there is no mail source specified, the mail gateway reads a single message from the standard input and submits the message to the roundup.mailgw module. UNIX mailbox: In this case, the gateway reads all messages from the UNIX mail spool file and submits each in turn to the roundup.mailgw module. The file is emptied once all messages have been successfully handled. The file is specified as:: mailbox /path/to/mailbox In all of the following mail source types, the username and password can be stored in a ``~/.netrc`` file. If done so, only the server name needs to be specified on the command-line. The username and/or password will be prompted for if not supplied on the command-line or in ``~/.netrc``. POP: For the mail source "pop", the gateway reads all messages from the POP server specified and submits each in turn to the roundup.mailgw module. The server is specified as:: pop username:password@server The username and password may be omitted:: pop username@server pop server are both valid. POPS: Connect to a POP server over tls/ssl. This supports the same notation as POP:: pops username:password@server APOP: Same as POP, but using Authenticated POP:: apop username:password@server IMAP: Connect to an IMAP server. This supports the same notation as that of POP mail:: imap username:password@server It also allows you to specify a specific mailbox other than INBOX using this format:: imap username:password@server mailbox IMAPS: Connect to an IMAP server over tls/ssl. This supports the same notation as IMAP:: imaps username:password@server [mailbox] IMAPS_CRAM: Connect to an IMAP server over tls/ssl using CRAM-MD5 authentication. This supports the same notation as IMAP:: imaps_cram username:password@server [mailbox] IMAPS_OAUTH: Connect to an IMAP server over tls/ssl using OAUTH authentication. Note that this does not support a password in imaps URLs. Instead it uses only the user and server and a command-line option for the directory with the files ``access_token``, ``refresh_token``, ``client_secret``, and ``client_id``. By default this directory is ``oauth`` in your tracker home directory. The access token is tried first and, if expired, the refresh token together with the client secret is used to retrieve a new access token. Note that both token files need to be *writeable*, the access token is continuously replaced and some cloud providers may also renew the refresh token from time to time:: imaps_oauth username@server [mailbox] The refresh and access tokens (the latter can be left empty), the client id and the client secret need to be retrieved via cloud provider specific protocols or websites. You need the requests_ library installed for OAuth. .. _requests: https://requests.readthedocs.io/en/latest/ .. index:: ! roundup-admin single: roundup-admin; usage single: roundup-admin; data formats single: roundup-admin; man page reference pair: roundup-admin; designator Using roundup-admin =================== Part of the installation includes a man page for roundup-admin. Ypu should be able to read it using ``man roundup-admin``. As shown above, it is a generic tool for manipulating the underlying database for you tracker. Examples above show how to use it to: * install and initialize a new tracker * export/import tracker data for migrating between backends * creating a new user fom the command line * list/find users in the tracker The basic usage is:: Usage: roundup-admin [options] [ ] Options: -i instance home -- specify the issue tracker "home directory" to administer -u -- the user[:password] to use for commands -d -- print full designators not just class id numbers -c -- when outputting lists of data, comma-separate them. Same as '-S ","'. -S -- when outputting lists of data, string-separate them -s -- when outputting lists of data, space-separate them. Same as '-S " "'. -V -- be verbose when importing -v -- report Roundup and Python versions (and quit) Only one of -s, -c or -S can be specified. Help: roundup-admin -h roundup-admin help -- this help roundup-admin help -- command-specific help roundup-admin help all -- all available help Commands: commit create classname property=value ... display designator[,designator]* export [[-]class[,class]] export_dir exporttables [[-]class[,class]] export_dir filter classname propname=value ... find classname propname=value ... genconfig get property designator[,designator]* help topic history designator [skipquiet] import import_dir importtables export_dir initialise [adminpw] install [template [backend [key=val[,key=val]]]] list classname [property] migrate pack period | date perftest [mode] [arguments]* pragma setting=value reindex [classname|designator]* restore designator[,designator]* retire designator[,designator]* rollback security [Role name] set items property=value property=value ... specification classname table classname [property[,property]*] templates [trace_search] updateconfig Commands may be abbreviated as long as the abbreviation matches only one command, e.g. l == li == lis == list. One thing to note, The ``-u user`` setting does not currently operate like a user logging in via the web. The user running roundup-admin must have read access to the tracker home directory. As a result the user has access to the files and the database info contained in config.ini. Using ``-u user`` sets the actor/user parameter in the journal. Changes that are made are attributed to that user. The password is ignored if provided. Any existing username has full access to the data just like the admin user. This is an area for further development so that roundup-admin could be used with sudo to provide secure command line access to a tracker. In general you should forget that there is a -u parameter. All commands (except help) require a tracker specifier. This is just the path to the roundup tracker you're working with. A roundup tracker is where roundup keeps the database and configuration file that defines an issue tracker. It may be thought of as the issue tracker's "home directory". It may be specified in the environment variable ``TRACKER_HOME`` or on the command line as "``-i tracker``". A designator is a classname and an itemid concatenated, eg. bug1, user10, ... Property values are represented as strings in command arguments and in the printed results: - Strings are, well, strings. - Password values will display as their encoded value. - Date values are printed in the full date format in the local time zone, and accepted in the full format or any of the partial formats explained below.:: Input of... Means... "2000-04-17.03:45" 2000-04-17.03:45:00 "2000-04-17" 2000-04-17.00:00:00 "01-25" yyyy-01-25.00:00:00 "08-13.22:13" yyyy-08-13.22:13:00 "11-07.09:32:43" yyyy-11-07.09:32:43 "14:25" yyyy-mm-dd.14:25:00 "8:47:11" yyyy-mm-dd.08:47:11 "2003" 2003-01-01.00:00:00 "2003-04" 2003-04-01.00:00:00 "." "right now" - Link values are printed as item designators. When given as an argument, item designators and key strings are both accepted. - Multilink values are printed as lists of item designators joined by commas. When given as an argument, item designators and key strings are both accepted; an empty string, a single item, or a list of items joined by commas is accepted. When multiple items are specified to the roundup get or roundup set commands, the specified properties are retrieved or set on all the listed items. When multiple results are returned by the roundup get or roundup find commands, they are printed one per line (default) or joined by commas (with the "``-c``" option). Where the command changes data, a login name/password is required. The login may be specified as either "``name``" or "``name:password``". - ``ROUNDUP_LOGIN`` environment variable - the "``-u``" command-line option If either the name or password is not supplied, they are obtained from the command-line. The ``-u user`` setting does not currently operate like a user logging in via the web. The user running roundup-admin must have read access to the tracker home directory. As a result the user has access to the files and the database info contained in config.ini. Using ``-u user`` sets the actor/user parameter in the journal. Changes that are made are attributed to that user. The password is ignored if provided. Any existing username has full access to the data just like the admin user. This is an area for further development so that roundup-admin could be used with sudo to provide secure command line access to a tracker. When you initialise a new tracker instance you are prompted for the admin password. If you want to initialise a tracker non-interactively you can put the initialise command and password on the command line. But this allows others on the host to see the password (using the ps command). To initialise a tracker non-interactively without exposing the password, create a file (e.g init_tracker) set to mode 600 (so only the owner can read it) with the contents: initialise admin_password and feed it to roundup-admin on standard input. E.G. cat init_tracker | roundup-admin -i tracker_dir (for more details see https://issues.roundup-tracker.org/issue2550789.) .. index:: ! roundup-admin; usage in scripts Using with the shell -------------------- With version 0.6.0 or newer of roundup (which introduced support for multiple designators to display and the -d, -S and -s flags): To find all messages regarding chatting issues that contain the word "spam", for example, you could execute the following command from the directory where the database dumps its files:: shell% for issue in `roundup-admin -ds find issue status=chatting`; do > grep -l spam `roundup-admin -ds ' ' get messages $issue` > done msg23 msg49 msg50 msg61 shell% Or, using the -dc option, this can be written as a single command:: shell% grep -l spam `roundup get messages \ \`roundup -dc find issue status=chatting\`` msg23 msg49 msg50 msg61 shell% You can also display issue contents:: shell% roundup-admin display `roundup-admin -dc get messages \ issue3,issue1` files: [] inreplyto: None recipients: [] author: 1 date: 2003-02-16.21:23:03 messageid: None summary: jkdskldjf files: [] inreplyto: None recipients: [] author: 1 date: 2003-02-15.01:59:11 messageid: None summary: jlkfjadsf or status:: shell% roundup-admin get name `/tools/roundup/bin/roundup-admin \ -dc -i /var/roundup/sysadmin get status issue3,issue1` unread deferred or status on a single line:: shell% echo `roundup-admin get name \`/tools/roundup/bin/roundup-admin \ -dc -i /var/roundup/sysadmin get status issue3,issue1\`` unread deferred which is the same as:: shell% roundup-admin -s get name `/tools/roundup/bin/roundup-admin \ -dc -i /var/roundup/sysadmin get status issue3,issue1` unread deferred Also the tautological:: shell% roundup-admin get name \ `roundup-admin -dc get status \`roundup-admin -dc find issue \ status=chatting\`` chatting chatting Remember the roundup commands that accept multiple designators accept them ',' separated so using '-dc' is almost always required. .. _`customisation documentation`: customizing.html .. _`upgrading documentation`: upgrading.html .. _`installation documentation`: installation.html code-1a221cf0758b/doc/announcement.txt0000644000000000000000000003440414427061170015650 0ustar 00000000000000I'm proud to release version 2.2.0 of the Roundup issue tracker. This release is a bugfix and minor feature release, so make sure to read `docs/upgrading.txt `_ to bring your tracker up to date. The changes, as usual, include some new features and many bug fixes. Note that you should run ``roundup-admin ... migrate`` to update the database schema version. Do this before you use the web, command-line or mail interface and before any users access the tracker. You can download it with:: pip download roundup then unpack and test/install the tarball. Also:: pip install roundup (preferably in a virtual environment) can be used. Among the notable improvements from the 2.1.0 release are: - Dynamic and static compression of http responses. This improves performance when a front end web server isn't serving compressed assets. - REST interface supports CORS allowing Roundup to be used by third party web sites. Can specify origins allowed to use the REST interface. OpenAPI (SWagger) docs can be added. Error handling/reporting improved. - Dockerfile to build a containerized Roundup instance. A docker-compose configuration to deploy a a mysql based tracker is also supplied. - New full text search methods. SQLite FTS and PostgreSQL full text search are supported. These allow search expressions in addition to simple word based searches. - Secret values in config.ini can be stored in external files. This allows config.ini to be stored in a VCS without exposing secrets. - Translation object added to internal database handle. This allows auditors and extensions to provide efficient translations. - MySQL database creation uses COLLATE utf8_general_ci - Wsgi startup improvements (must be enabled by setting feature flag). - Fix crash when importing legacy Roundup tracker with long integers. - Fix issues with Roundup unable to find supporting files when installed via pip. Removed additional references to distfiles module. The file CHANGES.txt has a detailed list of feature additions and bug fixes (57) for each release. The most recent changes from there are at the end of this announcement. Also see the information in doc/upgrading.txt. If you find bugs, please report them to issues AT roundup-tracker.org or create an account at https://issues.roundup-tracker.org and open a new ticket. If you have patches to fix the issues they can be attached to the email or uploaded to the tracker. Upgrading ========= If you're upgrading from an older version of Roundup you *must* follow all the "Software Upgrade" guidelines given in the doc/upgrading.txt documentation. Note that you should run ``roundup-admin ... migrate`` for all your trackers to update the database schema version. Do this before you use the web, command-line or mail interface and before any users access the tracker. Roundup requires Python 2 newer than version 2.7.2 or Python 3 newer than or equal to version 3.6 for correct operation. (Python 3.4 or 3.5 may work, but are not tested.) To give Roundup a try, just download (see below), unpack and run:: python demo.py then open the url printed by the demo app. Release info and download page: https://pypi.org/project/roundup Source and documentation is available at the website: https://www.roundup-tracker.org/ Mailing lists - the place to ask questions: https://sourceforge.net/p/roundup/mailman/ About Roundup ============= Roundup is a simple-to-use and install issue-tracking system with command-line, web and e-mail interfaces. It is based on the winning design from Ka-Ping Yee in the Software Carpentry "Track" design competition. Note: Ping is not responsible for this project. The contact for this project is rouilj at users.sourceforge.net. Use this address for security or other sensitive issues. Development discussions occur on the roundup-devel at lists.sourceforge.net mailing list. Tickets can be opened at https://issues.roundup-tracker.org. Roundup manages a number of issues (with flexible properties such as "description", "priority", and so on) and provides the ability to: (a) submit new issues, (b) find and edit existing issues, and (c) discuss issues with other participants. The system facilitates communication among the participants by managing discussions and notifying interested parties when issues are edited. One of the major design goals for Roundup that it be simple to get going. Roundup is therefore usable "out of the box" with any Python 2.7.2+ (or 3.6+) installation. It doesn't even need to be "installed" to be operational, though an install script is provided. It comes with five basic issue tracker templates * a classic bug/feature tracker * a more extensive devel tracker for bug/features etc. * a responsive version of the devel tracker * a jinja2 version of the devel template (work in progress) * a minimal skeleton and supports four database back-ends (anydbm, sqlite, mysql and postgresql). Recent Changes ============== From 2.1.0 to 2.2.0. Fixed: ------ - issue2551161 - Fix ResourceWarnings when running with -W default. Cleaned up leaking file descriptors from zopetal pre-compile, python module compile and loading localization file. (John Rouillard) - When using roundup-server with native SSL, only accept TLS v1.2. Previously it used to accept only TLS v1.1. 1.1 is deprecated by chrome. I don't expect this to be a major problem since a front end server (apache, Nginx...) is usually customer facing and terminates SSL. (John Rouillard) - Fix hang when valid user without authorization for REST tries to use the rest interface. (John Rouillard) - Remove Content-Type and make sure no content is returned by OPTIONS request in REST interface. (John Rouillard) - In write_html set the Content-Length when response is not encoded/compressed. (John Rouillard) - In REST interface do not raise UsageError for invalid api version. Return json error with proper message. Fixes crash. (John Rouillard) - In REST interface, allow extensions on URI less than 6 characters in length. All other paths with a . in then will be passed through without change. This allows items like a JWT to be passed as a path element. (John Rouillard) - issue2550995 - KeyError classic during roundup-admin install. Add paths to search for locale and template files. - issue2551167 - pip install in containerized environments puts template and locale files under site-packages where roundup can't find them. Change code to find them under site-packages. - REST replace hard coded list of child endpoints for /rest/ with list pulled from registered endpoints. So newly added endpoints are shown. (John Rouillard) - issue2551107 - Handle representation of long int in history params for python3. Causes SyntaxError crash when showing history due to long int e.g. 2345L. This is not a problem for roundup trackers created using 1.2.0 or newer. The fix may have predated the 1.2.0 release but where the fix actually landed (representing id as a string and not as an int) is unknown. - issue2551175 - Make ETag content-encoding aware. HTTP ETag headers now include a suffix indicating the content-encoding used to send the data per rfc7232. Properly validate any form of ETag suffixed or non-suffixed for If-Match. - issue2551178 - fix Traceback in Apache WSGI - during file upload - issue2551179 - make roundup-demo initialize templates using config_ini.ini overrides. Needed for jinja to set template lang etc. Recognize minimal template when presented with a full path. (John Kristensen (jerrykan) and John Rouillard) - handle configparser.InterpolationSyntaxError raised if value has a single %. Seems to afect python 3 only. Reported by nomicon on IRC. (John Rouillard) - add random delay to session database retry code between 0 and .125 seconds. This seems to help reduce stalled connections when a number of connections are made at the same time. Log remaining retries once 5 of them have been used. (John Rouillard) - issue2551169 - setup.py enters endless loop on gentoo linux python2 installation. Fixed. - issue2551185 - must set PYTHONPATH=... python2 setup.py install --prefix=/tmp/r2. Force insert --old-and-unmangable to get it to use a classic installer and not an easy install. This only affects python2. - issue2551186 - Python versions >= 3.3 no longer use socket.sslerror. Andrew (kragacles) patched uses of socket.sslerror in mailgy.py. Patch adapted to allow trapping sslerror under both python2 and 3. (John Rouillard) - issue2551142 - postgresql reworked to use savepoint/"rollback to" rather than commit()/rollback(). Using savepoint should be faster. - issue2551196 - Unset labelprop of a Multilink can lead to Python error when using context/history. (reported and initial patch: Nagy Gabor, John Rouillard) - Fix roundup-server to pass If-Range http header so Ranges work better. (John Rouillard) - issue2551183 - Replace references to distutils in roundup/dist/command (John Rouillard) - Fix hang if Range request was not able to be satified or a HEAD request was done. - Mark strings involved with password reset and registration for translation. (reported: Thomas Arendsen Hein, John Rouillard) - issue2551159 - cl.filter fails if filterspec is None (also group and sort). Passing a sort, group or filterprop param set to None to any filter() call should not cause a traceback. It will pretend as though no filter, sort or group was specified. (John Rouillard) - issue2551205 - Add support for specifying valid origins for api: xmlrpc/rest. Allows CORS to work with roundup backend. (John Rouillard) - new option added to config.ini: login_empty_passwords set to no by default. Setting this to yes allows a user with an empty password to login. - issue2551207 - Fix sorting by order attribute if order attributes can be None. Add a test. - issue2551203 fix CORS requests by providing proper headers and allowing unauthenticted CORS preflight requests. (Marcus Priesch and John Rouillard) - issue2551206 - removed some windows installer references that were missed. - document use of jinja2 templating as optional in config.ini file. Report if available or not. (John Rouillard) - make setup.py install the Zope and wsgi.py frontends under share/frontends. This matches the install of the cgi-bin/roundup.cgi frontend. (John Rouillard) - prevent submit button from showing up when using _generic.item.html if the user doesn't have edit permissions. (John Rouillard) - issue2551216 - create new mysql databases using COLLATE utf8_general_ci to prevent crashes in test suite. (John Rouillard) Features: --------- - issue2551147 - Enable compression of http responses in roundup. Allow roundup to return gzip, (br or zstd with added modules) Content-Encoded replies. Compression could be done in upstream proxies/wsgi server but this allows it to occur natively. (John Rouillard) - Change tracker templates adding required to login forms. Invokes browser error reporting if user forgets to fill in a field. (John Rouillard) - issue1596345 - filtering user list (need user.search.hml). Incorporate user search features from issues.roundup-tracker.org into classic template. Devel and responsive templates already have this feature. - issue2550917 - Add a: "Welcome user, you have logged in" ok_message on login. (Ashley Burke) - enable HTTP/1.1 for roundup-server. This enables keep-alive for faster response/loading. Also eliminates stalls when the front end web server uses http 1.1 but the roundup-server uses 1.0. New option "-V HTTP/1.0" can turn it off. (John Rouillard) - issue2551163 - add scripts/Docker/Dockerfile to provide basic support for containerization. See installation.txt for details. (John Rouillard) - issue2551163 - add scripts/Docker/docker-compose.yml to get a mysql/roundup deployment. (Norbert Schlemmer, modified by John Rouilard) - REST add openapi_doc decorator to add openapi_doc to endpoints. Decorate a couple of examples. (John Rouillard) - REST when incorrect method is used, report allowed methods in error message as well as in an Allow header. (John Rouillard) - REST change response to invalid attribute specified in path. Return 400 code not 405 code for this case and improve error. (John Rouillard) - REST correct values for some Access-Control-Allow-Methods and Access-Control-Allow-Headers headers. (John Rouillard) - issue2550991 - define default cache control settings for javascript and css assets. (John Rouillard) - issue2551181 - fragments can be appended to designators. So issue23#msg24 could jump to the element with id msg24 in issue 23. Before this patch you would have two links issue23 and msg24 separated by # (John Rouillard). - added small utility script to dump dbm based tracker databases (e.g. db/sessions). (John Rouillard) - issue2551182 - Enhance configuration module to allow loading values from an external file. Secrets (passwords, secrets) can specify file using file:// or file:///. The first line of the file is used as the secret. This allows committing config.ini to a VCS. (John Rouillard) - Added xapian indexer to Docker container. (John Rouillard) - Add support for indexer type native-fts to use FTS5 for sqlite databases. (John Rouillard) - Add support for indexer type native-fts to use PostreSQL's full text search. (John Rouillard) - Add better error display to the user. Needed to expose errors in fts5 search syntax to the user while also displaying the template page structure. (John Rouillard) - issue2551189 - increase size of words in full text index. Many terms (like exception names or symbolic constants) are larger than 25. Also German words are long. Since there is little chance of fixing German to shorten their words, change indexer maxlength to 50. (Thomas Arendsen Hein provided patch; patch reworked John Rouillard) - issue2551184 - add an i18n object to the roundupdb. This makes it possible to translate error messages in detectors (or actions). The i18n object is now also correctly set for the mail interface: previously the 'language' setting in the [mailgw] section seems to have been ignored. Thanks to Marcus Priesch for the patch. - issue2551212 - speed up wsgi interface by caching the tracker instance. Hidden behind a feature flag. See upgrading.txt for details. (Marcus Priesch with feature flag by John Rouillard) code-1a221cf0758b/doc/conf.py0000644000000000000000000002157114427061170013715 0ustar 00000000000000# -*- coding: utf-8 -*- # -- Roundup Notice -- # This conf.py is used to generate local documentation. There is also # website/www/conf.py in Roundup repository which uses the same .rst # sources, but different setup to generate site contents. # --- # Roundup documentation build configuration file, created by # sphinx-quickstart on Tue Jan 20 17:33:20 2009. # # This file is execfile()d with the current directory set to its # containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys import os # Read Roundup version by importing it from parent directory, # this ensures that 'unkown version' is inserted even if # `roundup` is importable from other location in sys.path SYSSAVE = sys.path DOCROOT = os.path.abspath(os.path.dirname(__file__)) sys.path = [os.path.dirname(DOCROOT)] try: from roundup import __version__ as VERSION SHORTVER = '.'.join(VERSION.split('.', 2)[:2]) except ImportError: VERSION = SHORTVER = '(unknown version)' finally: sys.path = SYSSAVE # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. #extensions = ['toctree'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # source_suffix = ['.rst', '.md'] source_suffix = '.txt' # The encoding of source files. #source_encoding = 'utf-8' # The master toctree document. master_doc = 'index' # General information about the project. project = u'Roundup' copyright = u'2009-2021, Richard Jones, Roundup-Team' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = SHORTVER # The full version, including alpha/beta/rc tags. release = VERSION # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # see https://issues.roundup-tracker.org/issue2550907: exclude_patterns = ['_build', 'announcement.txt', 'whatsnew-0.7.txt', 'whatsnew-0.8.txt'] # The reST default role (used for this markup: `text`) to use for all # documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. #keep_warnings = False # -- Options for HTML output ---------------------------------------------- #-- pre Sphinx 1.3 # The style sheet to use for HTML and HTML Help pages. A file of that name # must exist either in Sphinx' static/ path, or in one of the custom paths # given in html_static_path. html_style = 'style.css' #-- Sphinx 1.3 # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. #html_theme = 'alabaster' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied # directly to the root of the documentation. html_extra_path = ['html_extra'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Language to be used for generating the HTML full-text search index. # Sphinx supports the following languages: # 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' # 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' #html_search_language = 'en' # A dictionary with options for the search language support, empty by default. # Now only 'ja' uses this config value #html_search_options = {'type': 'default'} # The name of a javascript file (relative to the configuration directory) that # implements a search results scorer. If empty, the default will be used. #html_search_scorer = 'scorer.js' # Output file base name for HTML help builder. htmlhelp_basename = 'roundupdoc' # -- Options for LaTeX output --------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. #'preamble': '', # Latex figure (float) alignment #'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ ('docs/index', 'Roundup.tex', u'Roundup Documentation', u'Richard Jones', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True code-1a221cf0758b/doc/customizing.txt0000644000000000000000000072730414427061170015541 0ustar 00000000000000.. meta:: :description: How to customize and extend the Roundup Issue Tracker. Includes many cookbook and how-to examples. Reference for the design and internals needed to understand and extend the examples to meet new needs. :tocdepth: 2 =================== Customising Roundup =================== .. This document borrows from the ZopeBook section on ZPT. The original is at: http://www.zope.org/Documentation/Books/ZopeBook/current/ZPT.stx .. contents:: :depth: 2 :local: What You Can Do =============== Before you get too far, it's probably worth having a quick read of the Roundup `design documentation`_. Customisation of Roundup can take one of six forms: 1. `tracker configuration`_ changes 2. database, or `tracker schema`_ changes 3. "definition" class `database content`_ changes 4. behavioural changes through detectors_, extensions_ and interfaces.py_ 5. `security / access controls`_ 6. change the `web interface`_ The third case is special because it takes two distinctly different forms depending upon whether the tracker has been initialised or not. The other two may be done at any time, before or after tracker initialisation. Yes, this includes adding or removing properties from classes. Trackers in a Nutshell ====================== Trackers have the following structure: .. index:: single: tracker; structure db directory single: tracker; structure detectors directory single: tracker; structure extensions directory single: tracker; structure html directory single: tracker; structure html directory single: tracker; structure lib directory =================== ======================================================== Tracker File Description =================== ======================================================== config.ini Holds the basic `tracker configuration`_ schema.py Holds the `tracker schema`_ initial_data.py Holds any data to be entered into the database when the tracker is initialised (optional) interfaces.py Allows `modifying the core of Roundup`_ (optional) db/ Holds the tracker's database db/files/ Holds the tracker's upload files and messages db/backend_name Names the database back-end for the tracker (obsolete). Use the ``backend`` setting in the ``[rdbms]`` section of ``config.ini`` instead. detectors/ `Auditors and reactors`_ for this tracker extensions/ Additional `actions`_ and `templating utilities`_ html/ Web interface templates, images and style sheets lib/ optional common imports for detectors and extensions =================== ======================================================== .. index:: config.ini .. index:: configuration; see config.ini Tracker Configuration ===================== The ``config.ini`` located in your tracker home contains the basic configuration for the web and e-mail components of Roundup's interfaces. Changes to the data captured by your tracker is controlled by the `tracker schema`_. Some configuration is also performed using permissions - see the `security / access controls`_ section. For example, to allow users to automatically register through the email interface, you must grant the "Anonymous" Role the "Email Access" Permission. .. index:: single: config.ini; sections see: configuration; config.ini The following is taken from the `Python Library Reference`__ (July 18, 2018) section "ConfigParser -- Configuration file parser": The configuration file consists of sections, led by a [section] header and followed by name: value entries, with continuations in the style of RFC 822 (see section 3.1.1, “LONG HEADER FIELDS”); name=value is also accepted. Note that leading whitespace is removed from values. The optional values can contain format strings which refer to other values in the same section, or values in a special DEFAULT section. Additional defaults can be provided on initialization and retrieval. Lines beginning with '#' or ';' are ignored and may be used to provide comments. For example:: [My Section] foodir = %(dir)s/whatever dir = frob would resolve the "%(dir)s" to the value of "dir" ("frob" in this case) resulting in "foodir" being "frob/whatever". __ https://docs.python.org/2/library/configparser.html Example configuration settings are below. This is a partial list. Documentation on all the settings is included in the ``config.ini`` file. .. index:: config.ini; sections main Section **main** database -- ``db`` Database directory path. The path may be either absolute or relative to the directory containig this config file. templates -- ``html`` Path to the HTML templates directory. The path may be either absolute or relative to the directory containing this config file. static_files -- default *blank* A list of space separated directory paths (or a single directory). These directories hold additional static files available via Web UI. These directories may contain sitewide images, CSS stylesheets etc. If a '-' is included, the list processing ends and the TEMPLATES directory is not searched after the specified directories. If this option is not set, all static files are taken from the TEMPLATES directory. admin_email -- ``roundup-admin`` Email address that roundup will complain to if it runs into trouble. If the email address doesn't contain an ``@`` part, the MAIL_DOMAIN defined below is used. dispatcher_email -- ``roundup-admin`` The 'dispatcher' is a role that can get notified of new items to the database. It is used by the ERROR_MESSAGES_TO config setting. If the email address doesn't contain an ``@`` part, the MAIL_DOMAIN defined below is used. email_from_tag -- default *blank* Additional text to include in the "name" part of the From: address used in nosy messages. If the sending user is "Foo Bar", the From: line is usually: ``"Foo Bar" `` the EMAIL_FROM_TAG goes inside the "Foo Bar" quotes like so: ``"Foo Bar EMAIL_FROM_TAG" `` new_web_user_roles -- ``User`` Roles that a user gets when they register with Web User Interface. This is a comma-separated list of role names (e.g. ``Admin,User``). new_email_user_roles -- ``User`` Roles that a user gets when they register with Email Gateway. This is a comma-separated string of role names (e.g. ``Admin,User``). error_messages_to -- ``user`` Send error message emails to the ``dispatcher``, ``user``, or ``both``? The dispatcher is configured using the DISPATCHER_EMAIL setting. Allowed values: ``dispatcher``, ``user``, or ``both`` html_version -- ``html4`` HTML version to generate. The templates are ``html4`` by default. If you wish to make them xhtml, then you'll need to change this var to ``xhtml`` too so all auto-generated HTML is compliant. Allowed values: ``html4``, ``xhtml`` timezone -- ``0`` Numeric timezone offset used when users do not choose their own in their settings. instant_registration -- ``yes`` Register new users instantly, or require confirmation via email? Allowed values: ``yes``, ``no`` email_registration_confirmation -- ``yes`` Offer registration confirmation by email or only through the web? Allowed values: ``yes``, ``no`` indexer_stopwords -- default *blank* Additional stop-words for the full-text indexer specific to your tracker. See the indexer source for the default list of stop-words (e.g. ``A,AND,ARE,AS,AT,BE,BUT,BY, ...``). umask -- ``02`` Defines the file creation mode mask. csv_field_size -- ``131072`` Maximum size of a csv-field during import. Roundup's export format is a csv (comma separated values) variant. The csv reader has a limit on the size of individual fields starting with python 2.5. Set this to a higher value if you get the error 'Error: field larger than field limit' during import. .. index:: config.ini; sections tracker Section **tracker** name -- ``Roundup issue tracker`` A descriptive name for your Roundup instance. web -- ``http://host.example/demo/`` The web address that the tracker is viewable at. This will be included in information sent to users of the tracker. The URL MUST include the cgi-bin part or anything else that is required to get to the home page of the tracker. You MUST include a trailing '/' in the URL. email -- ``issue_tracker`` Email address that mail to Roundup should go to. language -- default *blank* Default locale name for this tracker. If this option is not set, the language is determined by the environment variable LANGUAGE, LC_ALL, LC_MESSAGES, or LANG, in that order of preference. .. index:: config.ini; sections web Section **web** allow_html_file -- ``no`` Setting this option enables Roundup to serve uploaded HTML file content *as HTML*. This is a potential security risk and is therefore disabled by default. Set to 'yes' if you trust *all* users uploading content to your tracker. http_auth -- ``yes`` Whether to use HTTP Basic Authentication, if present. Roundup will use either the REMOTE_USER or HTTP_AUTHORIZATION variables supplied by your web server (in that order). Set this option to 'no' if you do not wish to use HTTP Basic Authentication in your web interface. use_browser_language -- ``yes`` Whether to use HTTP Accept-Language, if present. Browsers send a language-region preference list. It's usually set in the client's browser or in their Operating System. Set this option to 'no' if you want to ignore it. debug -- ``no`` Setting this option makes Roundup display error tracebacks in the user's browser rather than emailing them to the tracker admin."), .. index:: config.ini; sections rdbms single: config.ini; database settings Section **rdbms** Settings in this section are used to set the backend and configure addition settings needed by RDBMs like SQLite, Postgresql and MySQL backends. .. index:: single: postgres; select backend in config.ini single: mysql; select backend in config.ini single: sqlite; select backend in config.ini single: anydbm; select backend in config.ini see: database; postgres see: database; mysql see: database; sqlite see: database; anydbm backend -- set to value by init The database backend such as anydbm, sqlite, mysql or postgres. name -- ``roundup`` Name of the database to use. host -- ``localhost`` Database server host. port -- default *blank* TCP port number of the database server. Postgresql usually resides on port 5432 (if any), for MySQL default port number is 3306. Leave this option empty to use backend default. user -- ``roundup`` Database user name that Roundup should use. password -- ``roundup`` Database user password. read_default_file -- ``~/.my.cnf`` Name of the MySQL defaults file. Only used in MySQL connections. read_default_group -- ``roundup`` Name of the group to use in the MySQL defaults file. Only used in MySQL connections. .. index:: single: sqlite; lock timeout sqlite_timeout -- ``30`` Number of seconds to wait when the SQLite database is locked. Used only for SQLite. cache_size -- `100` Size of the node cache (in elements) used to keep most recently used data in memory. .. index:: config.ini; sections logging see: logging; config.ini, sections logging Section **logging** config -- default *blank* Path to configuration file for standard Python logging module. If this option is set, logging configuration is loaded from specified file; options 'filename' and 'level' in this section are ignored. The path may be either absolute or relative to the directory containig this config file. filename -- default *blank* Log file name for minimal logging facility built into Roundup. If no file name specified, log messages are written on stderr. If above 'config' option is set, this option has no effect. The path may be either absolute or relative to the directory containig this config file. level -- ``ERROR`` Minimal severity level of messages written to log file. If above 'config' option is set, this option has no effect. Allowed values: ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR`` .. index:: config.ini; sections mail Section **mail** Outgoing email options. Used for nosy messages, password reset and registration approval requests. domain -- ``localhost`` Domain name used for email addresses. host -- default *blank* SMTP mail host that Roundup will use to send mail username -- default *blank* SMTP login name. Set this if your mail host requires authenticated access. If username is not empty, password (below) MUST be set! password -- default *blank* SMTP login password. Set this if your mail host requires authenticated access. port -- default *25* SMTP port on mail host. Set this if your mail host runs on a different port. local_hostname -- default *blank* The fully qualified domain name (FQDN) to use during SMTP sessions. If left blank, the underlying SMTP library will attempt to detect your FQDN. If your mail host requires something specific, specify the FQDN to use. tls -- ``no`` If your SMTP mail host provides or requires TLS (Transport Layer Security) then you may set this option to 'yes'. Allowed values: ``yes``, ``no`` tls_keyfile -- default *blank* If TLS is used, you may set this option to the name of a PEM formatted file that contains your private key. The path may be either absolute or relative to the directory containig this config file. tls_certfile -- default *blank* If TLS is used, you may set this option to the name of a PEM formatted certificate chain file. The path may be either absolute or relative to the directory containig this config file. charset -- utf-8 Character set to encode email headers with. We use utf-8 by default, as it's the most flexible. Some mail readers (eg. Eudora) can't cope with that, so you might need to specify a more limited character set (eg. iso-8859-1). debug -- default *blank* Setting this option makes Roundup to write all outgoing email messages to this file *instead* of sending them. This option has the same effect as environment variable SENDMAILDEBUG. Environment variable takes precedence. The path may be either absolute or relative to the directory containig this config file. add_authorinfo -- ``yes`` Add a line with author information at top of all messages send by Roundup. add_authoremail -- ``yes`` Add the mail address of the author to the author information at the top of all messages. If this is false but add_authorinfo is true, only the name of the actor is added which protects the mail address of the actor from being exposed at mail archives, etc. .. index:: config.ini; sections mailgw single: mailgw; config see: mail gateway; mailgw Section **mailgw** Roundup Mail Gateway options keep_quoted_text -- ``yes`` Keep email citations when accepting messages. Setting this to ``no`` strips out "quoted" text from the message. Signatures are also stripped. Allowed values: ``yes``, ``no`` leave_body_unchanged -- ``no`` Preserve the email body as is - that is, keep the citations *and* signatures. Allowed values: ``yes``, ``no`` default_class -- ``issue`` Default class to use in the mailgw if one isn't supplied in email subjects. To disable, leave the value blank. language -- default *blank* Default locale name for the tracker mail gateway. If this option is not set, mail gateway will use the language of the tracker instance. subject_prefix_parsing -- ``strict`` Controls the parsing of the [prefix] on subject lines in incoming emails. ``strict`` will return an error to the sender if the [prefix] is not recognised. ``loose`` will attempt to parse the [prefix] but just pass it through as part of the issue title if not recognised. ``none`` will always pass any [prefix] through as part of the issue title. subject_suffix_parsing -- ``strict`` Controls the parsing of the [suffix] on subject lines in incoming emails. ``strict`` will return an error to the sender if the [suffix] is not recognised. ``loose`` will attempt to parse the [suffix] but just pass it through as part of the issue title if not recognised. ``none`` will always pass any [suffix] through as part of the issue title. subject_suffix_delimiters -- ``[]`` Defines the brackets used for delimiting the commands suffix in a subject line. subject_content_match -- ``always`` Controls matching of the incoming email subject line against issue titles in the case where there is no designator [prefix]. ``never`` turns off matching. ``creation + interval`` or ``activity + interval`` will match an issue for the interval after the issue's creation or last activity. The interval is a standard Roundup interval. subject_updates_title -- ``yes`` Update issue title if incoming subject of email is different. Setting this to ``no`` will ignore the title part of the subject of incoming email messages. refwd_re -- ``(\s*\W?\s*(fw|fwd|re|aw|sv|ang)\W)+`` Regular expression matching a single reply or forward prefix prepended by the mailer. This is explicitly stripped from the subject during parsing. Value is Python Regular Expression (UTF8-encoded). origmsg_re -- `` ^[>|\s]*-----\s?Original Message\s?-----$`` Regular expression matching start of an original message if quoted in the body. Value is Python Regular Expression (UTF8-encoded). sign_re -- ``^[>|\s]*-- ?$`` Regular expression matching the start of a signature in the message body. Value is Python Regular Expression (UTF8-encoded). eol_re -- ``[\r\n]+`` Regular expression matching end of line. Value is Python Regular Expression (UTF8-encoded). blankline_re -- ``[\r\n]+\s*[\r\n]+`` Regular expression matching a blank line. Value is Python Regular Expression (UTF8-encoded). ignore_alternatives -- ``no`` When parsing incoming mails, Roundup uses the first text/plain part it finds. If this part is inside a multipart/alternative, and this option is set, all other parts of the multipart/alternative are ignored. The default is to keep all parts and attach them to the issue. .. index:: config.ini; sections php Section **pgp** OpenPGP mail processing options enable -- ``no`` Enable PGP processing. Requires gpg. roles -- default *blank* If specified, a comma-separated list of roles to perform PGP processing on. If not specified, it happens for all users. homedir -- default *blank* Location of PGP directory. Defaults to $HOME/.gnupg if not specified. .. index:: config.ini; sections nosy Section **nosy** Nosy messages sending messages_to_author -- ``no`` Send nosy messages to the author of the message. If ``yes`` is used, then messages are sent to the author even if not on the nosy list, same for ``new`` (but only for new messages). When set to ``nosy``, the nosy list controls sending messages to the author. Allowed values: ``yes``, ``no``, ``new``, ``nosy`` signature_position -- ``bottom`` Where to place the email signature. Allowed values: ``top``, ``bottom``, ``none`` add_author -- ``new`` Does the author of a message get placed on the nosy list automatically? If ``new`` is used, then the author will only be added when a message creates a new issue. If ``yes``, then the author will be added on followups too. If ``no``, they're never added to the nosy. Allowed values: ``yes``, ``no``, ``new`` add_recipients -- ``new`` Do the recipients (``To:``, ``Cc:``) of a message get placed on the nosy list? If ``new`` is used, then the recipients will only be added when a message creates a new issue. If ``yes``, then the recipients will be added on followups too. If ``no``, they're never added to the nosy. Allowed values: ``yes``, ``no``, ``new`` email_sending -- ``single`` Controls the email sending from the nosy reactor. If ``multiple`` then a separate email is sent to each recipient. If ``single`` then a single email is sent with each recipient as a CC address. max_attachment_size -- ``2147483647`` Attachments larger than the given number of bytes won't be attached to nosy mails. They will be replaced by a link to the tracker's download page for the file. .. index:: single: roundup-admin; config.ini update single: roundup-admin; config.ini create single: config.ini; create single: config.ini; update You may generate a new default config file using the ``roundup-admin genconfig`` command. You can generate a new config file merging in existing settings using the ``roundup-admin updateconfig`` command. Configuration variables may be referred to in lower or upper case. In code, variables not in the "main" section are referred to using their section and name, so "domain" in the section "mail" becomes MAIL_DOMAIN. .. index:: pair: configuration; extensions pair: configuration; detectors Extending the configuration file -------------------------------- You can't add new variables to the config.ini file in the tracker home but you can add two new config.ini files: - a config.ini in the ``extensions`` directory will be loaded and attached to the config variable as "ext". - a config.ini in the ``detectors`` directory will be loaded and attached to the config variable as "detectors". For example, the following in ``detectors/config.ini``:: [main] qa_recipients = email@example.com is accessible as:: db.config.detectors['QA_RECIPIENTS'] Note that the name grouping applied to the main configuration file is applied to the extension config files, so if you instead have:: [qa] recipients = email@example.com then the above ``db.config.detectors['QA_RECIPIENTS']`` will still work. Unlike values in the tracker's main ``config.ini``, the values defined in these config files are not validated. For example: a setting that is supposed to be an integer value (e.g. 4) could be the word "foo". If you are writing Python code that uses these settings, you should expect to handle invalid values. Also, incorrect values aren't discovered until the config setting is used. This can be long after the tracker is started and the error may not be seen in the logs. It is possible to validate these settings. Validation involves calling the ``update_options`` method on the configuration option. This can be done from the ``init()`` function in the Python files implementing extensions_ or detectors_. As an example, adding the following to an extension:: from roundup.configuration import SecretMandatoryOption def init(instance): instance.config.ext.update_option('RECAPTCHA_SECRET', SecretMandatoryOption,description="Secret securing reCaptcha.") similarly for a detector:: from roundup.configuration import MailAddressOption def init(db): try: db.config.detectors.update_option('QA_RECIPIENTS', MailAddressOption, description="Email used for QA comment followup.") except KeyError: # COMMENT_EMAIL setting is not found, but it's optional # so continue pass will allow reading the secret from a file or append the tracker domain to an email address if it does not have a domain. Running ``roundup-admin -i tracker_home display user1`` will validate the settings for both config.ini`s. Otherwise detector options are not validated until the first request to the web interface (or email gateway). There are 4 arguments for ``update_option``: 1. config setting name - string (positional, mandatory) 2. option type - Option derived class from configuration.py (positional, mandatory) 3. default value - string (optional, named default) 4. description - string (optional, named description) The first argument is the config setting name as described at the beginning of this section. The second argument is a class in the roundup.configuration module. There are a number of these classes: BooleanOption, IntegerNumberOption, RegExpOption.... Please see the configuration module for all Option validators and their descriptions. You can also define your own custom validator in `interfaces.py`_. The third and fourth arguments are strings and are optional. They are printed if there is an error and may help the user correct the problem. .. index:: ! schema Tracker Schema ============== .. note:: if you modify the schema, you'll most likely need to edit the `web interface`_ HTML template files and `detectors`_ to reflect your changes. A tracker schema defines what data is stored in the tracker's database. Schemas are defined using Python code in the ``schema.py`` module of your tracker. The ``schema.py`` and ``initial_data.py`` modules ------------------------------------------------- The schema.py module is used to define what your tracker looks like on the inside, the schema of the tracker. It defines the Classes and properties on each class. It also defines the security for those Classes. The next few sections describe how schemas work and what you can do with them. The initial_data.py module sets up the initial state of your tracker. It’s called exactly once - by the ``roundup-admin initialise`` command. See the start of the section on `database content`_ for more info about how this works. .. index:: schema; classic - description of The "classic" schema -------------------- The "classic" schema looks like this (see section `setkey(property)`_ below for the meaning of ``'setkey'`` -- you may also want to look into the sections `setlabelprop(property)`_ and `setorderprop(property)`_ for specifying (default) labelling and ordering of classes.):: pri = Class(db, "priority", name=String(), order=String()) pri.setkey("name") stat = Class(db, "status", name=String(), order=String()) stat.setkey("name") keyword = Class(db, "keyword", name=String()) keyword.setkey("name") user = Class(db, "user", username=String(), organisation=String(), password=String(), address=String(), realname=String(), phone=String(), alternate_addresses=String(), queries=Multilink('query'), roles=String(), timezone=String()) user.setkey("username") msg = FileClass(db, "msg", author=Link("user"), summary=String(), date=Date(), recipients=Multilink("user"), files=Multilink("file"), messageid=String(), inreplyto=String()) file = FileClass(db, "file", name=String()) issue = IssueClass(db, "issue", keyword=Multilink("keyword"), status=Link("status"), assignedto=Link("user"), priority=Link("priority")) issue.setkey('title') .. index:: schema; allowed changes What you can't do to the schema ------------------------------- You must never: **Remove the user class** This class is the only *required* class in Roundup. **Remove the "username", "address", "password" or "realname" user properties** Various parts of Roundup require these properties. Don't remove them. **Change the type of a property** Property types must *never* be changed - the database simply doesn't take this kind of action into account. Note that you can't just remove a property and re-add it as a new type either. If you wanted to make the assignedto property a Multilink, you'd need to create a new property assignedto_list and remove the old assignedto property. What you can do to the schema ----------------------------- Your schema may be changed at any time before or after the tracker has been initialised (or used). You may: **Add new properties to classes, or add whole new classes** This is painless and easy to do - there are generally no repercussions from adding new information to a tracker's schema. **Remove properties** Removing properties is a little more tricky - you need to make sure that the property is no longer used in the `web interface`_ *or* by the detectors_. Classes and Properties - creating a new information store --------------------------------------------------------- In the tracker above, we've defined 7 classes of information: priority Defines the possible levels of urgency for issues. status Defines the possible states of processing the issue may be in. keyword Initially empty, will hold keywords useful for searching issues. user Initially holding the "admin" user, will eventually have an entry for all users using Roundup. msg Initially empty, will hold all e-mail messages sent to or generated by Roundup. file Initially empty, will hold all files attached to issues. issue Initially empty, this is where the issue information is stored. We define the "priority" and "status" classes to allow two things: 1. reduction in the amount of information stored on the issue 2. more powerful, accurate searching of issues by priority and status By only requiring a link on the issue (which is stored as a single number) we reduce the chance that someone mis-types a priority or status - or simply makes a new one up. Class names are used to access items of that class in the `REST api`_ interface. The classic tracker was created before the REST interface was added. It uses the single form (i.e. issue and user not issues and users) for its classes. Most REST documentation suggests using plural forms. However, to make your API consistent, use singular forms for classes that you add. Class and Items ~~~~~~~~~~~~~~~ A Class defines a particular class (or type) of data that will be stored in the database. A class comprises one or more properties, which gives the information about the class items. The actual data entered into the database, using ``class.create()``, are called items. They have a special immutable property called ``'id'``. We sometimes refer to this as the *itemid*. .. index:: schema; property types Properties ~~~~~~~~~~ A Class is comprised of one or more properties of the following types: String properties are for storing arbitrary-length strings. Password properties are for storing encoded arbitrary-length strings. The default encoding is defined on the ``roundup.password.Password`` class. Date properties store date-and-time stamps. Their values are Timestamp objects. Interval properties store time periods rather than absolute dates. For example 2 hours. Integer properties store integer values. (Number can store real/float values.) Number properties store numeric values. There is an option to use double-precision floating point numbers. Boolean properties store on/off, yes/no, true/false values. Link properties refers to a single other item selected from a specified class. The class is part of the property; the value is an integer, the id of the chosen item. Multilink properties refer to possibly many items in a specified class. The value is a list of integers. Properties can have additional attributes to change the default behaviour: .. index:: triple: schema; property attributes; required triple: schema; property attributes; default_value triple: schema; property attributes; quiet * All properties support the following attributes: - ``required``: see `design documentation`_. Adds the property to the list returned by calling get_required_props for the class. - ``default_value``: see `design documentation`_ Sets the default value if the property is not set. - ``quiet``: see `design documentation`_. Suppresses user visible to changes to this property. The property change is not reported: - in the change feedback/confirmation message in the web interface - the property change section of the nosy email - the web history at the bottom of an item's page This can be used to store state of the user interface (e.g. the names of elements that are collapsed or hidden from the user). Making properties that are updated as an indirect result of a user's change (e.g. updating a blockers property, counting number of times an issue was reopened or reassigned etc.) should not be displayed to the user as they can be confusing. .. index:: triple: schema; property attributes; indexme * String properties can have an ``indexme`` attribute that defines if the property should be part of the full text index. The default is 'no' but this can be set to 'yes' to allow a property's contents to be in the full text index. .. index:: triple: schema; property attributes; use_double * Number properties can have a ``use_double`` attribute that, when set to ``True``, will use double precision floating point in the database. * Link and Multilink properties can have several attributes: .. index:: triple: schema; property attributes; do_journal - ``do_journal``: By default, every change of a link property is recorded in the item being linked to (or being unlinked). A typical use-case for setting ``do_journal='no'`` would be to turn off journalling of nosy list, message author and message recipient link and unlink events to prevent the journal from clogged with these events. .. index:: triple: schema; property attributes; try_id_parsing - ``try_id_parsing`` is turned on by default. If entering a number into a Link or Multilink field, Roundup interprets this number as an ID of the item to link to. Sometimes items can have numeric names (like, e.g., product codes). For these Roundup needs to match the numeric name and should never match an ID. In this case you can set ``try_id_parsing='no'``. .. index:: triple: schema; property attributes; rev_multilink - The ``rev_multilink`` option takes a property name to be inserted into the linked-to class. This property is a Multilink property that links back to the current class. The new Multilink is read-only (it is automatically modified if the Link or Multilink property defining it is modified). The new property can be used in normal searches using the "filter" method of the Class. This means it can be used like other Multilink properties when searching (in an index template) or via the REST and XMLRPC APIs. As a example, suppose you want to group multiple issues into a super issue. Each issue can be part of only one super issue. It is inefficient to find all of the issues that are part of the super issue by searching through all issues in the system looking at the part_of link property. To make this more efficient, you can declare an issue's part_of property as:: issue = IssueClass(db, "issue", ... part_of = Link("issue", rev_multilink="components"), ... ) This automatically creates the ``components`` multilink on the issue class. The ``components`` multilink is never explicitly declared in the issue class, but it has the same effect as though you had declared the class as:: issue = IssueClass(db, "issue", ... part_of = Link("issue"), components = Multilink("issue"), ... ) Then wrote a detector to update the components property on the corresponding issue. Writing this detector can be tricky. There is one other difference, you can not explicitly set/modify the ``components`` multilink. The effect of setting ``part_of = 3456`` on issue1234 automatically adds "1234" to the ``components`` property on issue3456. You can search the ``components`` multilink just like a regular multilink, but you can't explicitly assign to it. Another difference of reverse multilinks to normal multilinks is that when a linked node is retired, the node vanishes from the multilink, e.g. in the example above, if an issue with ``part_of`` set to another issue is retired this issue vanishes from the ``components`` multilink of the other issue. You can also link between different classes. So you can modify the issue definition to include:: issue = IssueClass(db, "issue", ... assigned_to = Link("user", rev_multilink="responsibleFor"), ... ) This makes it easy to list all issues that the user is responsible for (aka assigned_to). .. index:: triple: schema; property attributes; msg_header_property - The ``msg_header_property`` is used by the mail gateway when sending out messages. When a link or multilink property of an issue changes, Roundup creates email headers of the form:: X-Roundup-issue-prop: value where ``value`` is the ``name`` property for the linked item(s). For example, if you have a multilink for attached_files in your issue, you will see a header:: X-Roundup-issue-attached_files: MySpecialFile.doc, HisResume.txt when the class for attached files is defined as:: file = FileClass(db, "file", name=String()) ``MySpecialFile.doc`` is the name for the file object. If you have an ``assigned_to`` property in your issue class that links to the user class and you want to add a header:: X-Roundup-issue-assigned_to: ... so that the mail recipients can filter emails where ``X-Roundup-issue-assigned_to: name`` that contains their username. The user class is defined as:: user = Class(db, "user", username=String(), password=Password(), address=String(), realname=String(), phone=String(), organisation=String(), alternate_addresses=String(), queries=Multilink('query'), roles=String(), # comma-separated string of Role names timezone=String()) Because there is no ``name`` parameter for the user class, there will be no header. However setting:: assigned_to=Link("user", msg_header_property="username") will make the mail gateway generate an ``X-Roundup-issue-assigned_to`` using the username property of the linked user. Assume assigned_to for an issue is linked to the user with username=joe_user, setting:: msg_header_property="username" for the assigned_to property will generated message headers of the form:: X-Roundup-issue-assigned_to: joe_user for emails sent on issues where joe_user has been assigned to the issue. If this property is set to the empty string "", it will prevent the header from being generated on outgoing mail. .. index:: triple: schema; class property; creator triple: schema; class property; creation triple: schema; class property; actor triple: schema; class property; activity All Classes automatically have a number of properties by default: *creator* Link to the user that created the item. *creation* Date the item was created. *actor* Link to the user that last modified the item. *activity* Date the item was last modified. .. index:: triple: schema; class property; content triple: schema; class property; type FileClass ~~~~~~~~~ FileClasses save their "content" attribute off in a separate file from the rest of the database. This reduces the number of large entries in the database, which generally makes databases more efficient, and also allows us to use command-line tools to operate on the files. They are stored in the files sub-directory of the ``'db'`` directory in your tracker. FileClasses also have a "type" attribute to store the MIME type of the file. Roundup by default considers the contents of the file immutable. This is to assist in maintaining an accurate record of correspondence. The distributed tracker templates do not enforce this. So if you have access to the Roundup tracker directory, you can edit the files (make sure to preserve mode, owner and group) to remove information (e.g. if somebody includes a password or you need to redact proprietary information). Obviously the journal for the message/file will not report that the file has changed. Best practice is to remove offending material and leave a placeholder. E.G. replace a password with the text:: [password has been deleted 2020-12-02 --myname] If you need to delete an entire file, replace the file contents with:: [file contents deleted due to spam 2020-10-21 --myname] rather than deleting the file. If you actually delete the file Roundup will report an error to the user and email the administrator. If you empty the file, a user downloading the file using the direct URL (e.g. ``tracker/msg22``) may be confused and think something is broken when they receive an empty file. Retiring a file/msg does not prevent access to the file using the direct URL. Retiring an item only removes it when requesting a list of all items in the class. If you are replacing the contents, you probably want to change the content type of the file. E.G. from ``image/jpeg`` to ``text/plain``. You can do this easily through the web interface, or using the ``roundup-admin`` command line interface. You can also change the contents of a file or message using the REST interface. Note that this will NOT result in an entry in the journal, so again it allows a silent change. To do this you need to make two rest requests. An example using curl is:: $ curl -u demo:demo -s -H "X-requested-with: rest" \ -H "Referer: https://tracker.example.com/demo/" \ -X GET \ https://tracker.example.com/demo/rest/data/file/30/content { "data": { "id": "30", "type": "", "link": "https://tracker.example.com/demo/rest/data/file/30/content", "data": "hello3", "@etag": "\"3f2f8063dbce5b6bd43567e6f4f3c671\"" } } using the etag, overwrite the content with:: $ curl -u demo:demo -s -H "X-requested-with: rest" \ -H "Referer: https://tracker.example.com/demo/" \ -H 'If-Match: "3f2f8063dbce5b6bd43567e6f4f3c671"' \ -X PUT \ -F "data=@hello" \ https://tracker.example.com/demo/rest/data/file/30/content where ``hello`` is a file on local disk. You can enforce immutability in your tracker by adding an auditor (see detectors_) for the file/msg class that rejects changes to the content property. The auditor could also add a journal entry so that a change via the Roundup mechanism is reported. Using a mixin (see: https://wiki.roundup-tracker.org/MixinClassFileClass) to augment the file class allows for other possibilities including signing the file, or recording a checksum in the database and validating the file contents at the time it gets read. This allows detection of changes done on the filesystem outside of the Roundup mechanism. .. index:: triple: schema; class property; messages triple: schema; class property; files triple: schema; class property; nosy triple: schema; class property; superseder IssueClass ~~~~~~~~~~ IssueClasses automatically include the "messages", "files", "nosy", and "superseder" properties. The messages and files properties list the links to the messages and files related to the issue. The nosy property is a list of links to users who wish to be informed of changes to the issue - they get "CC'ed" e-mails when messages are sent to or generated by the issue. The nosy reactor (in the ``'detectors'`` directory) handles this action. The superseder link indicates an issue which has superseded this one. They also have the dynamically generated "creation", "activity" and "creator" properties. The value of the "creation" property is the date when an item was created, and the value of the "activity" property is the date when any property on the item was last edited (equivalently, these are the dates on the first and last records in the item's journal). The "creator" property holds a link to the user that created the issue. .. index: triple: schema; class method; setkey setkey(property) ~~~~~~~~~~~~~~~~ .. index:: roundup-admin; setting assignedto on an issue Select a String property of the class to be the key property. The key property must be unique, and allows references to the items in the class by the content of the key property. That is, we can refer to users by their username: for example, let's say that there's an issue in Roundup, issue 23. There's also a user, richard, who happens to be user 2. To assign an issue to him, we could do either of:: roundup-admin set issue23 assignedto=2 or:: roundup-admin set issue23 assignedto=richard Note, the same thing can be done in the web and e-mail interfaces. .. index: triple: schema; class method; setlabelprop setlabelprop(property) ~~~~~~~~~~~~~~~~~~~~~~ Select a property of the class to be the label property. The label property is used whereever an item should be uniquely identified, e.g., when displaying a link to an item. If setlabelprop is not specified for a class, the following values are tried for the label: * the key of the class (see the `setkey(property)`_ section above) * the "name" property * the "title" property * the first property from the sorted property name list So in most cases you can get away without specifying setlabelprop explicitly. You should make sure that users have View access to this property or the id property for a class. If the property can not be viewed by a user, looping over items in the class (e.g. messages attached to an issue) will not work. .. index: triple: schema; class method; setorderprop setorderprop(property) ~~~~~~~~~~~~~~~~~~~~~~ Select a property of the class to be the order property. The order property is used whenever using a default sort order for the class, e.g., when grouping or sorting class A by a link to class B in the user interface, the order property of class B is used for sorting. If setorderprop is not specified for a class, the following values are tried for the order property: * the property named "order" * the label property (see `setlabelprop(property)`_ above) So in most cases you can get away without specifying setorderprop explicitly. .. index: triple: schema; class method; create create(information) ~~~~~~~~~~~~~~~~~~~ Create an item in the database. This is generally used to create items in the "definitional" classes like "priority" and "status". .. index: schema; item ordering A note about ordering ~~~~~~~~~~~~~~~~~~~~~ When we sort items in the hyperdb, we use one of a number of methods, depending on the properties being sorted on: 1. If it's a String, Integer, Number, Date or Interval property, we just sort the scalar value of the property. Strings are sorted case-sensitively. 2. If it's a Link property, we sort by either the linked item's "order" property (if it has one) or the linked item's "id". 3. Mulitlinks sort similar to #2, but we start with the first Multilink list item, and if they're the same, we sort by the second item, and so on. Note that if an "order" property is defined on a Class that is used for sorting, all items of that Class *must* have a value against the "order" property, or sorting will result in random ordering. Examples of adding to your schema --------------------------------- Some examples are in the :ref:`CustomExamples` section below. Also you can start with `Roundup wiki CategorySchema`_ to see a list of additional examples of how schemas can be customised to add new functionality. .. _Roundup wiki CategorySchema: https://wiki.roundup-tracker.org/CategorySchema .. index:: !detectors .. _detectors: .. _Auditors and reactors: Detectors - adding behaviour to your tracker ============================================ Detectors are initialised every time you open your tracker database, so you're free to add and remove them any time, even after the database is initialised via the ``roundup-admin initialise`` command. The detectors in your tracker fire *before* (**auditors**) and *after* (**reactors**) changes to the contents of your database. They are Python modules that sit in your tracker's ``detectors`` directory. You will have some installed by default - have a look. You can write new detectors or modify the existing ones. The existing detectors installed for you are: .. index:: detectors; installed **nosyreaction.py** This provides the automatic nosy list maintenance and email sending. The nosy reactor (``nosyreaction``) fires when new messages are added to issues. The nosy auditor (``updatenosy``) fires when issues are changed, and figures out what changes need to be made to the nosy list (such as adding new authors, etc.) **statusauditor.py** This provides the ``chatty`` auditor which changes the issue status from ``unread`` or ``closed`` to ``chatting`` if new messages appear. It also provides the ``presetunread`` auditor which pre-sets the status to ``unread`` on new items if the status isn't explicitly defined. **messagesummary.py** Generates the ``summary`` property for new messages based on the message content. **userauditor.py** Verifies the content of some of the user fields (email addresses and roles lists). If you don't want this default behaviour, you're completely free to change or remove these detectors. See the detectors section in the `design document`__ for details of the interface for detectors. __ design.html .. index:: detectors; writing api Detector API ------------ .. index:: pair: detectors; auditors single: auditors; function signature single: auditors; defining single: auditors; arguments Auditors are called with the arguments:: audit(db, cl, itemid, newdata) where ``db`` is the database, ``cl`` is an instance of Class or IssueClass within the database, and ``newdata`` is a dictionary mapping property names to values. For a ``create()`` operation, the ``itemid`` argument is None and newdata contains all of the initial property values with which the item is about to be created. For a ``set()`` operation, newdata contains only the names and values of properties that are about to be changed. For a ``retire()`` or ``restore()`` operation, newdata is None. .. index:: pair: detectors; reactor single: reactors; function signature single: reactors; defining single: reactors; arguments Reactors are called with the arguments:: react(db, cl, itemid, olddata) where ``db`` is the database, ``cl`` is an instance of Class or IssueClass within the database, and ``olddata`` is a dictionary mapping property names to values. For a ``create()`` operation, the ``itemid`` argument is the id of the newly-created item and ``olddata`` is None. For a ``set()`` operation, ``olddata`` contains the names and previous values of properties that were changed. For a ``retire()`` or ``restore()`` operation, ``itemid`` is the id of the retired or restored item and ``olddata`` is None. .. index:: detectors; additional Additional Detectors Ready For Use ---------------------------------- Sample additional detectors that have been found useful will appear in the ``'detectors'`` directory of the Roundup distribution. If you want to use one, copy it to the ``'detectors'`` of your tracker instance: **irker.py** This detector sends notification on IRC through an irker daemon (http://www.catb.org/esr/irker/) when issues are created or messages are added. In order to use it you need to install irker, start the irkerd daemon, and add an ``[irker]`` section in ``detectors/config.ini`` that contains a comma-separated list of channels where the messages should be sent, e.g. ``channels = irc://chat.freenode.net/channelname``. **newissuecopy.py** This detector sends an email to a team address whenever a new issue is created. The address is hard-coded into the detector, so edit it before you use it (look for the text 'team@team.host') or you'll get email errors! **creator_resolution.py** Catch attempts to set the status to "resolved" - if the assignedto user isn't the creator, then set the status to "confirm-done". Note that "classic" Roundup doesn't have that status, so you'll have to add it. If you don't want to though, it'll just use "in-progress" instead. **email_auditor.py** If a file added to an issue is of type message/rfc822, we tack on the extension .eml. The reason for this is that Microsoft Internet Explorer will not open things with a .eml attachment, as they deem it 'unsafe'. Worse yet, they'll just give you an incomprehensible error message. For more information, see the detector code - it has a lengthy explanation. .. index:: auditors; rules for use single: reactors; rules for use Auditor or Reactor? ------------------- Generally speaking, the following rules should be observed: **Auditors** Are used for `vetoing creation of or changes to items`_. They might also make automatic changes to item properties. **Reactors** Detect changes in the database and react accordingly. They should avoid making changes to the database where possible, as this could create detector loops. Vetoing creation of or changes to items --------------------------------------- Auditors may raise the ``Reject`` exception to prevent the creation of or changes to items in the database. The mail gateway, for example, will not attach files or messages to issues when the creation of those files or messages are prevented through the ``Reject`` exception. It'll also not create users if that creation is ``Reject``'ed too. To use, simply add at the top of your auditor:: from roundup.exceptions import Reject And then when your rejection criteria have been detected, simply:: raise Reject('Description of error') Error messages raised with ``Reject`` automatically have any HTML content escaped before being displayed to the user. To display an error message to the user without performing any HTML escaping the ``RejectRaw`` should be used. All security implications should be carefully considering before using ``RejectRaw``. Generating email from Roundup ----------------------------- The module ``roundup.mailer`` contains most of the nuts-n-bolts required to generate email messages from Roundup. In addition, the ``IssueClass`` methods ``nosymessage()`` and ``send_message()`` are used to generate nosy messages, and may generate messages which only consist of a change note (ie. the message id parameter is not required - this is referred to as a "System Message" because it comes from "the system" and not a user). .. index:: extensions .. index:: extensions; add python functions to tracker .. _extensions: .. _actions: Extensions - adding capabilities to your tracker ================================================ While detectors_ add new behavior by reacting to changes in tracked objects, `extensions` add new actions and utilities to Roundup, which are mostly used to enhance web interface. You can create an extension by creating Python file in your tracker ``extensions`` directory. All files from this dir are loaded when tracker instance is created, at which point it calls ``init(instance)`` from each file supplying itself as a first argument. Note that at this point web interface is not loaded, but extensions still can register actions for in tracker instance. This may be fixed in Roundup 1.6 by introducing ``init_web(client)`` callback or a more flexible extension point mechanism. * ``instance.registerUtil`` is used for adding `templating utilities`_ (see `adding a time log to your issues`_ for an example) * ``instance.registerAction`` is used to add more actions to instance and to web interface. See `Defining new web actions`_ for details. Generic action can be added by inheriting from ``action.Action`` instead of ``cgi.action.Action``. .. _interfaces.py: .. _modifying the core of Roundup: interfaces.py - hooking into the core of Roundup ================================================ There is a magic trick for hooking into the core of Roundup. Using this you can: * modify class data structures * monkey patch core code to add new functionality * modify the email gateway * add new rest endpoints but with great power comes great responsibility. Interfaces.py has been around since the earliest releases of Roundup and used to be the main way to get a lot of customization done. In modern Roundup, the extensions_ mechanism is used, but there are places where interfaces.py is still useful. Note that the tracker directories are not on the Python system path when interfaces.py is evaluated. You need to add library directories explictly by modifying sys.path. Example: Changing Cache-Control headers --------------------------------------- The Client class in cgi/client.py has a lookup table that is used to set the Cache-Control headers for static files. The entries in this table are set from interfaces.py using:: from roundup.cgi.client import Client Client.Cache_Control['text/css'] = "public, max-age=3600" Client.Cache_Control['application/javascript'] = "public, max-age=30" Client.Cache_Control['rss.xml'] = "public, max-age=900" Client.Cache_Control['local.js'] = "public, max-age=7200" In this case static files delivered using @@file will have cache headers set. These files are searched for along the `static_files` path in the tracker's `config.ini`. In the example above: * a css file (e.g. @@file/style.css) will be cached for an hour * javascript files (e.g. @@file/libraries/jquery.js) will be cached for 30 seconds * a file named rss.xml will be cached for 15 minutes * a file named local.js will be cached for 2 hours Note that a file name match overrides the mime type settings. Example: Implement password complexity checking ----------------------------------------------- .. index:: tracker; lib directory (example) This example uses the zxcvbn_ module that you can place in the zxcvbn subdirectory of your tracker's lib directory. If you add this to the interfaces.py file in the root directory of your tracker (same place as schema.py):: import roundup.password as password from roundup.exceptions import Reject from zxcvbn import zxcvbn # monkey patch the setPassword method with this method # that checks password strength. origPasswordFunc = password.Password.setPassword def mpPasswordFunc(self, plaintext, scheme, config=None): """ Replace the password set function with one that verifies that the password is complex enough. It has to be done at this point and not in an auditor as the auditor only sees the encrypted password. """ results = zxcvbn(plaintext) if results['score'] < 3: l = [] map(l.extend, [[results['feedback']['warning']], results['feedback']['suggestions']]) errormsg = " ".join(l) raise Reject ("Password is too easy to guess. " + errormsg) return origPasswordFunc(self, plaintext, scheme, config=config) password.Password.setPassword = mpPasswordFunc it replaces the setPassword method in the Password class. The new version validates that the password is sufficiently complex. Then it passes off the setting of password to the original method. Example: Enhance time intervals ------------------------------- To make the user interface easier to use, you may want to support other forms for intervals. For example you can support an interval like 1.5 by interpreting it the same as 1:30 (1 hour 30 minutes). Also you can allow a bare integer (e.g. 45) as a number of minutes. To do this we intercept the from_raw method of the Interval class in hyperdb.py with:: import roundup.hyperdb as hyperdb origFrom_Raw = hyperdb.Interval.from_raw def normalizeperiod(self, value, **kw): ''' Convert alternate time forms into standard interval format [+-] [#y] [#m] [#w] [#d] [[[H]H:MM]:SS] if value is float, it's hour and fractional hours if value is integer, it's number of minutes ''' if ":" not in value: # Not a specified interval # if int consider number of minutes try: isMinutes = int(value) minutes = isMinutes%60 hours = (isMinutes - minutes) / 60 value = "%d:%d"%(hours,minutes) except ValueError: pass # if float, consider it number of hours and fractional hours. import math try: afterdecimal, beforedecimal = math.modf(float(value)) value = "%d:%d"%(beforedecimal,60*afterdecimal) except ValueError: pass return origFrom_Raw(self, value, **kw) hyperdb.Interval.from_raw = normalizeperiod any call to convert an interval from raw form now has two simpler (and more friendly) ways to specify common time intervals. Example: Modifying the mail gateway ----------------------------------- One site receives email on a main gateway. The virtual alias delivery table on the postfix server is configured with:: test-issues@example.com roundup-test@roundup-vm.example.com test-support@example.com roundup-test+support-a@roundup-vm.example.com test@support.example.com roundup-test+support-b@roundup-vm.example.com These modifications to the mail gateway for Roundup allows anonymous submissions. It hides all of the requesters under the "support" user. It also makes some other modifications to the mail parser allowing keywords to be set and prefixes to be defined based on the delivery alias. This is the entry in interfaces.py:: import roundup.mailgw import email.utils class SupportTracker(object): def __init__(self, prefix=None, keyword=None): self.prefix = prefix self.keyword = keyword # Define new prefixes and keywords based on local address. support_trackers = { ### production instances ### ### test instances ### 'roundup-test+support-a': SupportTracker(prefix='Support 1', keyword='support1'), 'roundup-test+support-b': SupportTracker(prefix='Support 2', keyword='support2'), 'roundup-test2+support-a': SupportTracker(prefix='Support 1', keyword='support1'), 'roundup-test2+support-b': SupportTracker(prefix='Support 2', keyword='support2'), } class parsedMessage(roundup.mailgw.parsedMessage): def __init__(self, mailgw, message, support_tracker): roundup.mailgw.parsedMessage.__init__(self, mailgw, message) if support_tracker.prefix: self.prefix = '%s: ' % support_tracker.prefix else: self.prefix = '' self.keywords = [] if support_tracker.keyword: try: self.keywords = [ self.db.keyword.lookup(support_tracker.keyword)] except KeyError: pass self.config.ADD_AUTHOR_TO_NOSY = 'no' self.config.ADD_RECIPIENTS_TO_NOSY = 'no' self.config.MAILGW_KEEP_QUOTED_TEXT = 'yes' self.config.MAILGW_LEAVE_BODY_UNCHANGED = 'yes' self.classname = 'issue' self.pfxmode = 'loose' self.sfxmode = 'none' # set the support user id self.fixed_author = self.db.user.lookup('support') self.fixed_props = { 'nosy': [self.fixed_author], 'keyword': self.keywords, } def handle_help(self): pass def check_subject(self): if not self.subject: self.subject = 'no subject' def rego_confirm(self): pass def get_author_id(self): # force the support user to be the author self.author = self.fixed_author def get_props(self): self.props = {} if not self.nodeid: self.props.update(self.fixed_props) self.props['title'] = ("%s%s" % ( self.prefix, self.subject.replace('[', '(').replace(']', ')'))) def get_content_and_attachments(self): roundup.mailgw.parsedMessage.get_content_and_attachments(self) if not self.content: self.content = 'no text' intro = [] for header in ['From', 'To', 'Cc']: for addr in self.message.getaddrlist(header): intro.append('%s: %s' % (header, email.utils.formataddr(addr))) intro.append('Subject: %s' % self.subject) intro.append('\n') self.content = '\n'.join(intro) + self.content class MailGW(roundup.mailgw.MailGW): def parsed_message_class(self, mailgw, message): support_tracker = None # The delivered-to header is unique to postfix # it is the target address: # roundup-test+support-a@roundup-vm.example.com # rather than # test-support@example.com recipients = message.getaddrlist('delivered-to') if recipients: localpart = recipients[0][1].rpartition('@')[0] support_tracker = support_trackers.get(localpart) if support_tracker: # parse the mesage using the parsedMessage class # defined above. return parsedMessage(mailgw, message, support_tracker) else: # parse the message normally return roundup.mailgw.parsedMessage(mailgw, message) This is the most complex example section. The mail gateway is also one of the more complex subsystems in Roundup, and modifying it is not trivial. Other Examples -------------- See the `rest interface documentation`_ for instructions on how to add new rest endpoints or `change the rate limiting method`_ using interfaces.py. Database Content ================ .. note:: If you modify the content of definitional classes, you'll most likely need to edit the tracker `detectors`_ to reflect your changes. Customisation of the special "definitional" classes (eg. status, priority, resolution, ...) may be done either before or after the tracker is initialised. The actual method of doing so is completely different in each case though, so be careful to use the right one. **Changing content before tracker initialisation** Edit the initial_data.py module in your tracker to alter the items created using the ``create( ... )`` methods. **Changing content after tracker initialisation** As the "admin" user, click on the "class list" link in the web interface to bring up a list of all database classes. Click on the name of the class you wish to change the content of. You may also use the ``roundup-admin`` interface's create, set and retire methods to add, alter or remove items from the classes in question. See "`adding a new field to the classic schema`_" for an example that requires database content changes. Security / Access Controls ========================== A set of Permissions is built into the security module by default: - Create (everything) - Edit (everything) - Search (everything) (used if View does not permit access) - View (everything) - Register (User class only) These are assigned to the "Admin" Role by default, and allow a user to do anything. Every Class you define in your `tracker schema`_ also gets an Create, Edit and View Permission of its own. The web and email interfaces also define: *Email Access* If defined, the user may use the email interface. Used by default to deny Anonymous users access to the email interface. When granted to the Anonymous user, they will be automatically registered by the email interface (see also the ``new_email_user_roles`` configuration option). *Web Access* If defined, the user may use the web interface. All users are able to see the login form, regardless of this setting (thus enabling logging in). *Web Roles* Controls user access to editing the "roles" property of the "user" class. TODO: deprecate in favour of a property-based control. *Rest Access* and *Xmlrpc Access* These control access to the Rest and Xmlrpc endpoints. The Admin and User roles have these by default in the classic tracker. See the `directions in the rest interface documentation`_ and the `xmlrpc interface documentation`_. These are hooked into the default Roles: - Admin (Create, Edit, Search, View and everything; Web Roles) - User (Web Access; Email Access) - Anonymous (Web Access) And finally, the "admin" user gets the "Admin" Role, and the "anonymous" user gets "Anonymous" assigned when the tracker is installed. For the "User" Role, the "classic" tracker defines: - Create, Edit and View issue, file, msg, query, keyword - View priority, status - View user - Edit their own user record And the "Anonymous" Role is defined as: - Web interface access - Register user (for registration) - View issue, file, msg, query, keyword, priority, status Put together, these settings appear in the tracker's ``schema.py`` file:: # # TRACKER SECURITY SETTINGS # # See the configuration and customisation document for information # about security setup. # # REGULAR USERS # # Give the regular users access to the web and email interface db.security.addPermissionToRole('User', 'Web Access') db.security.addPermissionToRole('User', 'Email Access') db.security.addPermissionToRole('User', 'Rest Access') db.security.addPermissionToRole('User', 'Xmlrpc Access') # Assign the access and edit Permissions for issue, file and message # to regular users now for cl in 'issue', 'file', 'msg', 'keyword': db.security.addPermissionToRole('User', 'View', cl) db.security.addPermissionToRole('User', 'Edit', cl) db.security.addPermissionToRole('User', 'Create', cl) for cl in 'priority', 'status': db.security.addPermissionToRole('User', 'View', cl) # May users view other user information? Comment these lines out # if you don't want them to p = db.security.addPermission(name='View', klass='user', properties=('id', 'organisation', 'phone', 'realname', 'timezone', 'username')) db.security.addPermissionToRole('User', p) # Users should be able to edit their own details -- this permission is # limited to only the situation where the Viewed or Edited item is their own. def own_record(db, userid, itemid, **ctx): '''Determine whether the userid matches the item being accessed.''' return userid == itemid p = db.security.addPermission(name='View', klass='user', check=own_record, description="User is allowed to view their own user details") db.security.addPermissionToRole('User', p) p = db.security.addPermission(name='Edit', klass='user', check=own_record, properties=('username', 'password', 'address', 'realname', 'phone', 'organisation', 'alternate_addresses', 'queries', 'timezone'), description="User is allowed to edit their own user details") db.security.addPermissionToRole('User', p) # Users should be able to edit and view their own queries. They should also # be able to view any marked as not private. They should not be able to # edit others' queries, even if they're not private def view_query(db, userid, itemid): private_for = db.query.get(itemid, 'private_for') if not private_for: return True return userid == private_for def edit_query(db, userid, itemid): return userid == db.query.get(itemid, 'creator') p = db.security.addPermission(name='View', klass='query', check=view_query, description="User is allowed to view their own and public queries") db.security.addPermissionToRole('User', p) p = db.security.addPermission(name='Search', klass='query') db.security.addPermissionToRole('User', p) p = db.security.addPermission(name='Edit', klass='query', check=edit_query, description="User is allowed to edit their queries") db.security.addPermissionToRole('User', p) p = db.security.addPermission(name='Retire', klass='query', check=edit_query, description="User is allowed to retire their queries") db.security.addPermissionToRole('User', p) p = db.security.addPermission(name='Restore', klass='query', check=edit_query, description="User is allowed to restore their queries") db.security.addPermissionToRole('User', p) p = db.security.addPermission(name='Create', klass='query', description="User is allowed to create queries") db.security.addPermissionToRole('User', p) # # ANONYMOUS USER PERMISSIONS # # Let anonymous users access the web interface. Note that almost all # trackers will need this Permission. The only situation where it's not # required is in a tracker that uses an HTTP Basic Authenticated front-end. db.security.addPermissionToRole('Anonymous', 'Web Access') # Let anonymous users access the email interface (note that this implies # that they will be registered automatically, hence they will need the # "Create" user Permission below) # This is disabled by default to stop spam from auto-registering users on # public trackers. #db.security.addPermissionToRole('Anonymous', 'Email Access') # Assign the appropriate permissions to the anonymous user's Anonymous # Role. Choices here are: # - Allow anonymous users to register db.security.addPermissionToRole('Anonymous', 'Register', 'user') # Allow anonymous users access to view issues (and the related, linked # information) for cl in 'issue', 'file', 'msg', 'keyword', 'priority', 'status': db.security.addPermissionToRole('Anonymous', 'View', cl) # Allow the anonymous user to use the "Show Unassigned" search. # It acts like "Show Open" if this permission is not available. # If you are running a tracker that does not allow read access for # anonymous, you should remove this entry as it can be used to perform # a username guessing attack against a roundup install. p = db.security.addPermission(name='Search', klass='user') db.security.addPermissionToRole ('Anonymous', p) # [OPTIONAL] # Allow anonymous users access to create or edit "issue" items (and the # related file and message items) #for cl in 'issue', 'file', 'msg': # db.security.addPermissionToRole('Anonymous', 'Create', cl) # db.security.addPermissionToRole('Anonymous', 'Edit', cl) .. index:: single: roundup-admin; view class permissions You can use ``roundup-admin security`` to verify the permissions defined in the schema. It also verifies that properties specified in permissions are valid for the class. This helps detect typos that can cause baffling permission issues. Automatic Permission Checks --------------------------- Permissions are automatically checked when information is rendered through the web. This includes: 1. View checks for properties when being rendered via the ``plain()`` or similar methods. If the check fails, the text "[hidden]" will be displayed. 2. Edit checks for properties when the edit field is being rendered via the ``field()`` or similar methods. If the check fails, the property will be rendered via the ``plain()`` method (see point 1. for subsequent checking performed) 3. View checks are performed in index pages for each item being displayed such that if the user does not have permission, the row is not rendered. 4. View checks are performed at the top of item pages for the Item being displayed. If the user does not have permission, the text "You are not allowed to view this page." will be displayed. 5. View checks are performed at the top of index pages for the Class being displayed. If the user does not have permission, the text "You are not allowed to view this page." will be displayed. New User Roles -------------- New users are assigned the Roles defined in the config file as: - NEW_WEB_USER_ROLES - NEW_EMAIL_USER_ROLES The `users may only edit their issues`_ example shows customisation of these parameters. Changing Access Controls ------------------------ You may alter the configuration variables to change the Role that new web or email users get, for example to not give them access to the web interface if they register through email. You may use the ``roundup-admin`` "``security``" command to display the current Role and Permission configuration in your tracker. Adding a new Permission ~~~~~~~~~~~~~~~~~~~~~~~ When adding a new Permission, you will need to: 1. add it to your tracker's ``schema.py`` so it is created, using ``security.addPermission``, for example:: db.security.addPermission(name="View", klass='frozzle', description="User is allowed to access frozzles") will set up a new "View" permission on the Class "frozzle". 2. enable it for the Roles that should have it (verify with "``roundup-admin security``") 3. add it to the relevant HTML interface templates 4. add it to the appropriate xxxPermission methods on in your tracker interfaces module The ``addPermission`` method takes a few optional parameters: **properties** A sequence of property names that are the only properties to apply the new Permission to (eg. ``... klass='user', properties=('name', 'email') ...``) **props_only** A boolean value (set to false by default) that is a new feature in Roundup 1.6. A permission defined using: ``properties=('list', 'of', 'property', 'names')`` is used to determine access for things other than just those properties. For example a check for View permission on the issue class or an issue item can use any View permission for the issue class even if that permission has a property list. This can be confusing and surprising as you would think that a permission including properties would be used only for determining the access permission for those properties. ``roundup-admin security`` will report invalid properties for the class. For example a permission with an invalid summary property is presented as:: Allowed to see content of object regardless of spam status (View for "file": ('content', 'summary') only) **Invalid properties for file: ['summary'] Setting ``props_only=True`` will make the permission valid only for those properties. If you use a lot of permissions with property checks, it can be difficult to change all of them. Calling the function: db.security.set_props_only_default(True) at the top of ``schema.py`` will make every permission creation behave as though props_only was set to True. It is expected that the default of True will become the default in a future Roundup release. **check** A function to be executed which returns boolean determining whether the Permission is allowed. If it returns True, the permission is allowed, if it returns False the permission is denied. The function can have one of two signatures:: check(db, userid, itemid) or:: check(db, userid, itemid, **ctx) where ``db`` is a handle on the open database, ``userid`` is the user attempting access and ``itemid`` is the specific item being accessed. If the second form is used the ``ctx`` dictionary is defined with the following values:: ctx['property'] the name of the property being checked or None if it's a class check. ctx['classname'] the name of the class that is being checked (issue, query ....). ctx['permission'] the name of the permission (e.g. View, Edit...). The second form is preferred as it makes it easier to implement more complex permission schemes. An example of the use of ``ctx`` can be found in the ``upgrading.txt`` or `upgrading.html`_ document. .. _`upgrading.html`: upgrading.html Example Scenarios ~~~~~~~~~~~~~~~~~ See the `examples`_ section for longer examples of customisation. **anonymous access through the e-mail gateway** Give the "anonymous" user the "Email Access", ("Edit", "issue") and ("Create", "msg") Permissions but do not not give them the ("Create", "user") Permission. This means that when an unknown user sends email into the tracker, they're automatically logged in as "anonymous". Since they don't have the ("Create", "user") Permission, they won't be automatically registered, but since "anonymous" has permission to use the gateway, they'll still be able to submit issues. Note that the Sender information - their email address - will not be available - they're *anonymous*. **automatic registration of users in the e-mail gateway** By giving the "anonymous" user the ("Register", "user") Permission, any unidentified user will automatically be registered with the tracker (with no password, so they won't be able to log in through the web until an admin sets their password). By default new Roundup trackers don't allow this as it opens them up to spam. It may be enabled by uncommenting the appropriate addPermissionToRole in your tracker's ``schema.py`` file. The new user is given the Roles list defined in the "new_email_user_roles" config variable. **only developers may be assigned issues** Create a new Permission called "Fixer" for the "issue" class. Create a new Role "Developer" which has that Permission, and assign that to the appropriate users. Filter the list of users available in the assignedto list to include only those users. Enforce the Permission with an auditor. See the example `restricting the list of users that are assignable to a task`_. **only managers may sign off issues as complete** Create a new Permission called "Closer" for the "issue" class. Create a new Role "Manager" which has that Permission, and assign that to the appropriate users. In your web interface, only display the "resolved" issue state option when the user has the "Closer" Permissions. Enforce the Permission with an auditor. This is very similar to the previous example, except that the web interface check would look like:: **don't give web access to users who register through email** Create a new Role called "Email User" which has all the Permissions of the normal "User" Role minus the "Web Access" Permission. This will allow users to send in emails to the tracker, but not access the web interface. **let some users edit the details of all users** Create a new Role called "User Admin" which has the Permission for editing users:: db.security.addRole(name='User Admin', description='Managing users') p = db.security.getPermission('Edit', 'user') db.security.addPermissionToRole('User Admin', p) and assign the Role to the users who need the permission. Web Interface ============= .. contents:: :local: The web interface is provided by the ``roundup.cgi.client`` module and is used by ``roundup.cgi``, ``roundup-server`` and ``ZRoundup`` (``ZRoundup`` is broken, until further notice). In all cases, we determine which tracker is being accessed (the first part of the URL path inside the scope of the CGI handler) and pass control on to the ``roundup.cgi.client.Client`` class - which handles the rest of the access through its ``main()`` method. This means that you can do pretty much anything you want as a web interface to your tracker. Repercussions of changing the tracker schema --------------------------------------------- If you choose to change the `tracker schema`_ you will need to ensure the web interface knows about it: 1. Index, item and search pages for the relevant classes may need to have properties added or removed, 2. The "page" template may require links to be changed, as might the "home" page's content arguments. How requests are processed -------------------------- The basic processing of a web request proceeds as follows: 1. figure out who we are, defaulting to the "anonymous" user 2. figure out what the request is for - we call this the "context" 3. handle any requested action (item edit, search, ...) 4. render the template requested by the context, resulting in HTML output In some situations, exceptions occur: - HTTP Redirect (generally raised by an action) - SendFile (generally raised by ``determine_context``) here we serve up a FileClass "content" property - SendStaticFile (generally raised by ``determine_context``) here we serve up a file from the tracker "html" directory - Unauthorised (generally raised by an action) here the action is cancelled, the request is rendered and an error message is displayed indicating that permission was not granted for the action to take place - NotFound (raised wherever it needs to be) this exception percolates up to the CGI interface that called the client Roundup URL design ------------------ Each tracker has several hardcoded URLs. These three are equivalent and lead to the main tracker page: 1. ``/`` 2. ``/index`` 3. ``/home`` The following prefix is used to access static resources: 4. ``/@@file/`` Two additional url's are used for the API's. The `REST api`_ is accessed via: 5. ``/rest/`` .. _`REST api`: rest.html and the `XMLRPC api`_ is available at: 6. ``/xmlrpc`` .. _`XMLRPC api`: xmlrpc.html All other URLs depend on the classes configured in Roundup database. Each class receives two URLs - one for the class itself and another for specific items of that class. Example for class URL: 7. ``/issue`` This is usually used to show listings of class items. The URL for for specific object of issue class with id 1 will look like: 8. ``/issue1`` .. _strip_zeros: Note that a leading string of 0's will be stripped from the id part of the object designator in the URL. E.G. ``/issue001`` is the same as ``/issue1``. Similarly for ``/file01`` etc. However you should generate URL's without the extra zeros. Determining web context ----------------------- To determine the "context" of a request (what request is for), we look at the URL path after the tracker root and at ``@template`` request parameter. Typical URL paths look like: 1. ``/tracker/issue`` 2. ``/tracker/issue1`` 3. ``/tracker/@@file/style.css`` 4. ``/cgi-bin/roundup.cgi/tracker/file1`` 5. ``/cgi-bin/roundup.cgi/tracker/file1/kitten.png`` where tracker root is ``/tracker/`` or ``/cgi-bin/roundup.cgi/tracker/`` We're looking at "issue", "issue1", "@@file/style.css", "file1" and "file1/kitten.png" in the cases above. 1. with is no path we are in the "home" context. See `the "home" context`_ below for details. "index" or "home" paths may also be used to switch into "home" context. 2. for paths starting with "@@file" the additional path entry ("style.css" in the example above) specifies the static file to be served from the tracker TEMPLATES directory (or STATIC_FILES, if configured). This is usually the tracker's "html" directory. Internally this works by raising SendStaticFile exception. 3. if there is something in the path (as in example 1, "issue"), it identifies the tracker class to display. 4. if the path is an item designator (as in examples 2 and 4, "issue1" and "file1"), then we're to display a specific item. :ref:`Note. ` 5. if the path starts with an item designator and is longer than one entry (as in example 5, "file1/kitten.png"), then we're assumed to be handling an item of a ``FileClass``, and the extra path information gives the filename that the client is going to label the download with (i.e. "file1/kitten.png" is nicer to download than "file1"). This raises a ``SendFile`` exception. Neither 2. or 5. use templates and stop before the template is determined. For other contexts the template used is specified by the ``@template`` variable, which defaults to: - only classname supplied: "index" - full item designator supplied: "item" The "home" Context ------------------ The "home" context is special because it allows you to add templated pages to your tracker that don't rely on a class or item (ie. an issues list or specific issue). Let's say you wish to add frames to control the layout of your tracker's interface. You'd probably have: - A top-level frameset page. This page probably wouldn't be templated, so it could be served as a static file (see `serving static content`_) - A sidebar frame that is templated. Let's call this page "home.navigation.html" in your tracker's "html" directory. To load that page up, you use the URL: /home?@template=navigation Serving static content ---------------------- See the previous section `determining web context`_ where it describes ``@@file`` paths. These files are served without any permission checks. Any user on the internet with the url can download the file. This is rarely an issue since the html templates are just source code and much of it can be found in the Roundup repository. Other decoration (logos, stylesheets) are similarly not security sensitive. You can use the static_files setting in config.ini to eliminate access to the templates directory if desired. If a file resolves to a symbolic link, it is not served. Performing actions in web requests ---------------------------------- When a user requests a web page, they may optionally also request for an action to take place. As described in `how requests are processed`_, the action is performed before the requested page is generated. Actions are triggered by using a ``@action`` CGI variable, where the value is one of: **login** Attempt to log a user in. **logout** Log the user out - make them "anonymous". **register** Attempt to create a new user based on the contents of the form and then log them in. **edit** Perform an edit of an item in the database. There are some `special form variables`_ you may use. Also you can set the ``__redirect_to`` form variable to the URL that should be displayed after the edit is succesfully completed. If you wanted to edit a sequence of issues, users etc. this could be used to display the next item in the sequence to the user. **new** Add a new item to the database. You may use the same `special form variables`_ as in the "edit" action. Also you can set the ``__redirect_to`` form variable to the URL that should be displayed after the new item is created. This is useful if you want to create another item rather than edit the newly created item. **retire** Retire the item in the database. **editCSV** Performs an edit of all of a class' items in one go. See also the *class*.csv templating method which generates the CSV data to be edited, and the ``'_generic.index'`` template which uses both of these features. **search** Mangle some of the form variables: - Set the form ":filter" variable based on the values of the filter variables - if they're set to anything other than "dontcare" then add them to :filter. - Also handle the ":queryname" variable and save off the query to the user's query list. Each of the actions is implemented by a corresponding ``*XxxAction*`` (where "Xxx" is the name of the action) class in the ``roundup.cgi.actions`` module. These classes are registered with ``roundup.cgi.client.Client``. If you need to define new actions, you may add them there (see `defining new web actions`_). Each action class also has a ``*permission*`` method which determines whether the action is permissible given the current user. The base permission checks for each action are: **login** Determine whether the user has the "Web Access" Permission. **logout** No permission checks are made. **register** Determine whether the user has the ("Create", "user") Permission. **edit** Determine whether the user has permission to edit this item. If we're editing the "user" class, users are allowed to edit their own details - unless they try to edit the "roles" property, which requires the special Permission "Web Roles". **new** Determine whether the user has permission to create this item. No additional property checks are made. Additionally, new user items may be created if the user has the ("Create", "user") Permission. **editCSV** Determine whether the user has permission to edit this class. **search** Determine whether the user has permission to view this class. Protecting users from web application attacks --------------------------------------------- There is a class of attacks known as Cross Site Request Forgeries (CSRF). Malicious code running in the browser can making a request to Roundup while you are logged into Roundup. The malicious code piggy backs on your existing Roundup session to make changes without your knowledge. Roundup 1.6 has support for defending against this by analyzing the * Referer, * Origin, and * Host or * X-Forwarded-Host HTTP headers. It compares the headers to the value of the web setting in the [tracker] section of the tracker's ``config.ini``. Also a per form token (also called a nonce) can be enabled for the tracker using the ``csrf_enforce_token`` option in config.ini. When enabled, Roundup will validate a hidden form field called ``@csrf``. If the validation fails (or the token is used more than once) the request is rejected. The ``@csrf`` input field is added automatically by calling the ``submit`` function/path. It can also be added manually by calling anti_csrf_nonce() directly. For example:: By default a nonce lifetime is 2 weeks. However the lifetime (in minutes) can be set by passing a lifetime argument as shown above. The example above makes the nonce lifetime 10 minutes. Search for @csrf in this document for more examples. There are more examples and information in ``upgrading.txt``. The token protects you because malicious code supplied by another site is unable to obtain the token. Thus many attempts they make to submit a request are rejected. The protection on the xmlrpc interface is untested, but is based on a valid header check against the Roundup url and the presence of the ``X-REQUESTED-WITH`` header. Work to improve this is a future project after the 1.6 release. The enforcement levels can be modified in ``config.ini``. Refer to that file for details. Special form variables ---------------------- Item properties and their values are edited with html FORM variables and their values. You can: - Change the value of some property of the current item. - Create a new item of any class, and edit the new item's properties, - Attach newly created items to a multilink property of the current item. - Remove items from a multilink property of the current item. - Specify that some properties are required for the edit operation to be successful. - Redirect to a different page after creating a new item (new action only, not edit action). Usually you end up on the page for the created item. - Set up user interface locale. These operations will only take place if the form action (the ``@action`` variable) is "edit" or "new". In the following, values are variable, "@" may be either ":" or "@", and other text "required" is fixed. Two special form variables are used to specify user language preferences: ``@language`` value may be locale name or ``none``. If this variable is set to locale name, web interface language is changed to given value (provided that appropriate translation is available), the value is stored in the browser cookie and will be used for all following requests. If value is ``none`` the cookie is removed and the language is changed to the tracker default, set up in the tracker configuration or OS environment. ``@charset`` value may be character set name or ``none``. Character set name is stored in the browser cookie and sets output encoding for all HTML pages generated by Roundup. If value is ``none`` the cookie is removed and HTML output is reset to Roundup internal encoding (UTF-8). Most properties are specified as form variables: ```` property on the current context item ``"@"`` property on the indicated item (for editing related information) Designators name a specific item of a class. ```` Name an existing item of class . ``"-"`` Name the th new item of class . If the form submission is successful, a new item of is created. Within the submitted form, a particular designator of this form always refers to the same new item. Once we have determined the "propname", we look at it to see if it's special: ``@required`` The associated form value is a comma-separated list of property names that must be specified when the form is submitted for the edit operation to succeed. When the is missing, the properties are for the current context item. When is present, they are for the item specified by . The "@required" specifier must come before any of the properties it refers to are assigned in the form. ``@remove@=id(s)`` or ``@add@=id(s)`` The "@add@" and "@remove@" edit actions apply only to Multilink properties. The form value must be a comma-separate list of keys for the class specified by the simple form variable. The listed items are added to (respectively, removed from) the specified property. ``@link@=`` If the edit action is "@link@", the simple form variable must specify a Link or Multilink property. The form value is a comma-separated list of designators. The item corresponding to each designator is linked to the property given by simple form variable. None of the above (ie. just a simple form value) The value of the form variable is converted appropriately, depending on the type of the property. For a Link('klass') property, the form value is a single key for 'klass', where the key field is specified in schema.py. For a Multilink('klass') property, the form value is a comma-separated list of keys for 'klass', where the key field is specified in schema.py. Note that for simple-form-variables specifiying Link and Multilink properties, the linked-to class must have a key field. For a String() property specifying a filename, the file named by the form value is uploaded. This means we try to set additional properties "filename" and "type" (if they are valid for the class). Otherwise, the property is set to the form value. For Date(), Interval(), Boolean(), Integer() and Number() properties, the form value is converted to the appropriate value. Any of the form variables may be prefixed with a classname or designator. Setting the form variable: ``__redirect_to=`` to a url when @action=new redirects the user to the specified url after successfully creating the new item. This is useful if you want the user to create another item rather than edit the newly created item. Note that the url assigned to ``__redirect_to`` must be url encoded/quoted and be under the tracker's base url. If the base_url uses http, you can set the url to https. Two special form values are supported for backwards compatibility: @note This is equivalent to:: @link@messages=msg-1 msg-1@content=value which is equivalent to the html:: except that in addition, the "author" and "date" properties of "msg-1" are set to the userid of the submitter, and the current time, respectively. @file This is equivalent to:: @link@files=file-1 file-1@content=value by adding the HTML:: The String content value is handled as described above for file uploads. If both the "@note" and "@file" form variables are specified, the action:: msg-1@link@files=file-1 is also performed. This would be expressed in HTML with:: We also check that FileClass items have a "content" property with actual content, otherwise we remove them from all_props before returning. Default templates ----------------- The default templates are html4 compliant. If you wish to change them to be xhtml compliant, you'll need to change the ``html_version`` configuration variable in ``config.ini`` to ``'xhtml'`` instead of ``'html4'``. Most customisation of the web view can be done by modifying the templates in the tracker ``'html'`` directory. There are several types of files in there. The *minimal* template includes: **page.html** This template usually defines the overall look of your tracker. When you view an issue, it appears inside this template. When you view an index, it also appears inside this template. This template defines a macro called "icing" which is used by almost all other templates as a coating for their content, using its "content" slot. It also defines the "head_title" and "body_title" slots to allow setting of the page title. **home.html** the default page displayed when no other page is indicated by the user **home.classlist.html** a special version of the default page that lists the classes in the tracker **classname.item.html** displays an item of the *classname* class **classname.index.html** displays a list of *classname* items **classname.search.html** displays a search page for *classname* items **_generic.index.html** used to display a list of items where there is no ``*classname*.index`` available **_generic.help.html** used to display a "class help" page where there is no ``*classname*.help`` **user.register.html** a special page just for the user class, that renders the registration page **style.css** a static file that is served up as-is The *classic* template has a number of additional templates. Remember that you can create any template extension you want to, so if you just want to play around with the templating for new issues, you can copy the current "issue.item" template to "issue.test", and then access the test template using the "@template" URL argument:: http://your.tracker.example/tracker/issue?@template=test and it won't affect your users using the "issue.item" template. You can also put templates into a subdirectory of the template directory. So if you specify:: http://your.tracker.example/tracker/issue?@template=test/item you will use the template at: ``test/issue.item.html``. If that template doesn't exit it will try to use ``test/_generic.item.html``. If that template doesn't exist it will return an error. Implementing Modal Editing Using @template ------------------------------------------ Many item templates allow you to edit the item. They contain code that renders edit boxes if the user has edit permissions. Otherwise the template will just display the item information. In some cases you want to do a modal edit. The user has to take some action (click a button or follow a link) to shift from display mode to edit mode. When the changes are submitted, ending the edit mode, the user is returned to display mode. Modal workflows usually slow things down and are not implemented by default templates. However for some workflows a modal edit is useful. For example a batch edit mode that allows the user to edit a number of issues all from one form could be implemented as a modal workflow of: * search for issues to modify * switch to edit mode and change values * exit back to the results of the search To implement the modal edit, assume you have an issue.edit.html template that implements an edit form. On the display page (a version of issue.item.html modified to only display information) add a link that calls the display url, but adds ``@template=edit`` to the link. This will now display the edit page. On the edit page you want to add a hidden text field to your form named ``@template`` with the value: ``item|edit``. When the form is submitted it is validated. If the form is correct the user will see the item rendered using the item template. If there is an error (validation failed) the item will be rendered using the edit template. The edit template that is rendered will display all the changes that the user made to the form before it was submitted. The user can correct the error and resubmit the changes until the form validates. If the form failed to validate but the ``@template`` field had the value ``item`` the user would still see the error, but all of the data the user entered would be discarded. The user would have to redo all the edits again. How the templates work ---------------------- Templating engines ~~~~~~~~~~~~~~~~~~ Since version 1.4.20 Roundup supports two templating engines: * the original `Template Attribute Language`_ (TAL) engine from Zope * the standalone Chameleon templating engine. Chameleon is intended as a replacement for the original TAL engine, and supports the same syntax, but they are not 100% compatible. The major (and most likely the only) incompatibility is the default expression type being ``python:`` instead of ``path:``. See also "Incompatibilities and differences" section of `Chameleon documentation`__. Version 1.5.0 added experimental support for the `jinja2`_ templating language. You must install the `jinja2`_ module in order to use it. The ``jinja2`` template supplied with Roundup has the templates rewritten to use ``jinja2`` rather than TAL. A number of trackers are running using ``jinja2`` templating so it is considered less experimental than Chameleon templating. .. _jinja2: https://palletsprojects.com/p/jinja/ **NOTE1**: For historical reasons, examples given below assumes path expression as default expression type. With Chameleon you have to manually resolve the path expressions. A Chameleon-based, z3c.pt, that is fully compatible with the old TAL implementation, is planned to be included in a future release. **NOTE2**: As of 1.4.20 Chameleon support is highly experimental and **not** recommended for production use. .. _Chameleon: https://pypi.org/project/Chameleon/ .. _z3c.pt: https://pypi.org/project/z3c.pt/ __ https://chameleon.readthedocs.io/en/latest/reference.html?highlight=differences#incompatibilities-and-differences .. _TAL: .. _Template Attribute Language: https://pagetemplates.readthedocs.io/en/latest/history/TALSpecification14.html Basic Templating Actions ~~~~~~~~~~~~~~~~~~~~~~~~ Roundup's templates consist of special attributes on the HTML tags. These attributes form the **Template Attribute Language**, or TAL. The basic TAL commands are: **tal:define="variable expression; variable expression; ..."** Define a new variable that is local to this tag and its contents. For example:: In this example, the variable "title" is defined as the result of the expression "request/description". The "tal:content" command inside the tag may then use the "title" variable. **tal:condition="expression"** Only keep this tag and its contents if the expression is true. For example::

Display some issue information.

In the example, the

tag and its contents are only displayed if the user has the "View" permission for issues. We consider the number zero, a blank string, an empty list, and the built-in variable nothing to be false values. Nearly every other value is true, including non-zero numbers, and strings with anything in them (even spaces!). **tal:repeat="variable expression"** Repeat this tag and its contents for each element of the sequence that the expression returns, defining a new local variable and a special "repeat" variable for each element. For example:: The example would iterate over the sequence of users returned by "user/list" and define the local variable "u" for each entry. Using the repeat command creates a new variable called "repeat" which you may access to gather information about the iteration. See the section below on `the repeat variable`_. **tal:replace="expression"** Replace this tag with the result of the expression. For example:: The example would replace the tag and its contents with the user's realname. If the user's realname was "Bruce", then the resultant output would be "Bruce". **tal:content="expression"** Replace the contents of this tag with the result of the expression. For example:: user's name appears here The example would replace the contents of the tag with the user's realname. If the user's realname was "Bruce" then the resultant output would be "Bruce". **tal:attributes="attribute expression; attribute expression; ..."** Set attributes on this tag to the results of expressions. For example:: My Details In the example, the "href" attribute of the tag is set to the value of the "string:user${request/user/id}" expression, which will be something like "user123". **tal:omit-tag="expression"** Remove this tag (but not its contents) if the expression is true. For example:: Hello, world! would result in output of:: Hello, world! Note that the commands on a given tag are evaulated in the order above, so *define* comes before *condition*, and so on. Additionally, you may include tags such as , which are removed from output. Its content is kept, but the tag itself is not (so don't go using any "tal:attributes" commands on it). This is useful for making arbitrary blocks of HTML conditional or repeatable (very handy for repeating multiple table rows, which would otherwise require an illegal tag placement to effect the repeat). Templating Expressions ~~~~~~~~~~~~~~~~~~~~~~ Templating Expressions are covered by `Template Attribute Language Expression Syntax`_, or TALES. The expressions you may use in the attribute values may be one of the following forms: **Path Expressions** - eg. ``item/status/checklist`` These are object attribute / item accesses. Roughly speaking, the path ``item/status/checklist`` is broken into parts ``item``, ``status`` and ``checklist``. The ``item`` part is the root of the expression. We then look for a ``status`` attribute on ``item``, or failing that, a ``status`` item (as in ``item['status']``). If that fails, the path expression fails. When we get to the end, the object we're left with is evaluated to get a string - if it is a method, it is called; if it is an object, it is stringified. Path expressions may have an optional ``path:`` prefix, but they are the default expression type, so it's not necessary. If an expression evaluates to ``default``, then the expression is "cancelled" - whatever HTML already exists in the template will remain (tag content in the case of ``tal:content``, attributes in the case of ``tal:attributes``). If an expression evaluates to ``nothing`` then the target of the expression is removed (tag content in the case of ``tal:content``, attributes in the case of ``tal:attributes`` and the tag itself in the case of ``tal:replace``). If an element in the path may not exist, then you can use the ``|`` operator in the expression to provide an alternative. So, the expression ``request/form/foo/value | default`` would simply leave the current HTML in place if the "foo" form variable doesn't exist. You may use the python function ``path``, as in ``path("item/status")``, to embed path expressions in Python expressions. **String Expressions** - eg. ``string:hello ${user/name}`` These expressions are simple string interpolations - though they can be just plain strings with no interpolation if you want. The expression in the ``${ ... }`` is just a path expression as above. **Python Expressions** - eg. ``python: 1+1`` These expressions give the full power of Python. All the "root level" variables are available, so ``python:item.status.checklist()`` would be equivalent to ``item/status/checklist``, assuming that ``checklist`` is a method. Modifiers: **structure** - eg. ``structure python:msg.content.plain(hyperlink=1)`` The result of expressions are normally *escaped* to be safe for HTML display (all "<", ">" and "&" are turned into special entities). The ``structure`` expression modifier turns off this escaping - the result of the expression is now assumed to be HTML, which is passed to the web browser for rendering. **not:** - eg. ``not:python:1=1`` This simply inverts the logical true/false value of another expression. .. _TALES: .. _Template Attribute Language Expression Syntax: https://pagetemplates.readthedocs.io/en/latest/history/TALESSpecification13.html Template Macros ~~~~~~~~~~~~~~~ Macros are used in Roundup to save us from repeating the same common page stuctures over and over. The most common (and probably only) macro you'll use is the "icing" macro defined in the "page" template. Macros are generated and used inside your templates using special attributes similar to the `basic templating actions`_. In this case, though, the attributes belong to the `Macro Expansion Template Attribute Language`_, or METAL. The macro commands are: **metal:define-macro="macro name"** Define that the tag and its contents are now a macro that may be inserted into other templates using the *use-macro* command. For example:: ... defines a macro called "page" using the ```` tag and its contents. Once defined, macros are stored on the template they're defined on in the ``macros`` attribute. You can access them later on through the ``templates`` variable, eg. the most common ``templates/page/macros/icing`` to access the "page" macro of the "page" template. **metal:use-macro="path expression"** Use a macro, which is identified by the path expression (see above). This will replace the current tag with the identified macro contents. For example:: ... will replace the tag and its contents with the "page" macro of the "page" template. **metal:define-slot="slot name"** and **metal:fill-slot="slot name"** To define *dynamic* parts of the macro, you define "slots" which may be filled when the macro is used with a *use-macro* command. For example, the ``templates/page/macros/icing`` macro defines a slot like so:: title goes here In your *use-macro* command, you may now use a *fill-slot* command like this:: My Title where the tag that fills the slot completely replaces the one defined as the slot in the macro. Note that you may not mix `METAL`_ and `TAL`_ commands on the same tag, but TAL commands may be used freely inside METAL-using tags (so your *fill-slots* tags may have all manner of TAL inside them). .. _METAL: .. _Macro Expansion Template Attribute Language: https://pagetemplates.readthedocs.io/en/latest/history/TALESSpecification13.html Information available to templates ---------------------------------- This is implemented by ``roundup.cgi.templating.RoundupPageTemplate`` The following variables are available to templates. **context** The current context. This is either None, a `hyperdb class wrapper`_ or a `hyperdb item wrapper`_ **request** Includes information about the current request, including: - the current index information (``filterspec``, ``filter`` args, ``properties``, etc) parsed out of the form. - methods for easy filterspec link generation - "form" The current CGI form information as a mapping of form argument name to value (specifically a cgi.FieldStorage) - "env" the CGI environment variables - "base" the base URL for this instance - "user" a HTMLItem instance for the current user - "language" as determined by the browser or config - "classname" the current classname (possibly None) - "template" the current template (suffix, also possibly None) **config** This variable holds all the values defined in the tracker config.ini file (eg. TRACKER_NAME, etc.) **db** The current database, used to access arbitrary database items. **templates** Access to all the tracker templates by name. Used mainly in *use-macro* commands. **utils** This variable makes available some utility functions like batching. **nothing** This is a special variable - if an expression evaluates to this, then the tag (in the case of a ``tal:replace``), its contents (in the case of ``tal:content``) or some attributes (in the case of ``tal:attributes``) will not appear in the the output. So, for example:: Hello, World! would result in:: Hello, World! **default** Also a special variable - if an expression evaluates to this, then the existing HTML in the template will not be replaced or removed, it will remain. So:: Hello, World! would result in:: Hello, World! **true**, **false** Boolean constants that may be used in `templating expressions`_ instead of ``python:1`` and ``python:0``. **i18n** Internationalization service, providing two string translation methods: **gettext** (*message*) Return the localized translation of message **ngettext** (*singular*, *plural*, *number*) Like ``gettext()``, but consider plural forms. If a translation is found, apply the plural formula to *number*, and return the resulting message (some languages have more than two plural forms). If no translation is found, return singular if *number* is 1; return plural otherwise. The context variable ~~~~~~~~~~~~~~~~~~~~ The *context* variable is one of three things based on the current context (see `determining web context`_ for how we figure this out): 1. if we're looking at a "home" page, then it's None 2. if we're looking at a specific hyperdb class, it's a `hyperdb class wrapper`_. 3. if we're looking at a specific hyperdb item, it's a `hyperdb item wrapper`_. If the context is not None, we can access the properties of the class or item. The only real difference between cases 2 and 3 above are: 1. the properties may have a real value behind them, and this will appear if the property is displayed through ``context/property`` or ``context/property/field``. 2. the context's "id" property will be a false value in the second case, but a real, or true value in the third. Thus we can determine whether we're looking at a real item from the hyperdb by testing "context/id". Hyperdb class wrapper ::::::::::::::::::::: This is implemented by the ``roundup.cgi.templating.HTMLClass`` class. This wrapper object provides access to a hyperdb class. It is used primarily in both index view and new item views, but it's also usable anywhere else that you wish to access information about a class, or the items of a class, when you don't have a specific item of that class in mind. We allow access to properties. There will be no "id" property. The value accessed through the property will be the current value of the same name from the CGI form. There are several methods available on these wrapper objects: =========== ============================================================= Method Description =========== ============================================================= properties return a `hyperdb property wrapper`_ for all of this class's properties that are searchable by the user. You can use the argument cansearch=False to get all properties. list lists all of the active (not retired) items in the class. csv return the items of this class as a chunk of CSV text. propnames lists the names of the properties of this class. filter lists of items from this class, filtered and sorted. Two options are available for sorting: 1. by the current *request* filterspec/filter/sort/group args 2. by the "filterspec", "sort" and "group" keyword args. "filterspec" is ``{propname: value(s)}``. "sort" and "group" are an optionally empty list ``[(dir, prop)]`` where dir is '+', '-' or None and prop is a prop name or None. The propname in filterspec and prop in a sort/group spec may be transitive, i.e., it may contain properties of the form link.link.link.name. eg. All issues with a priority of "1" with messages added in the last week, sorted by activity date: ``issue.filter(filterspec={"priority": "1", 'messages.creation' : '.-1w;'}, sort=[('activity', '+')])`` Note that when searching for Link and Multilink values, the special value '-1' searches for empty Link or Multilink values. For both, Links and Multilinks, multiple values given in a filter call are combined with 'OR' by default. For Multilinks a postfix expression syntax using negative ID numbers (as strings) as operators is supported. Each non-negative number (or '-1') is pushed on an operand stack. A negative number pops the required number of arguments from the stack, applies the operator, and pushes the result. The following operators are supported: - '-2' stands for 'NOT' and takes one argument - '-3' stands for 'AND' and takes two arguments - '-4' stands for 'OR' and takes two arguments Note that this special handling of ID arguments is applied only when a negative number smaller than -1 is encountered as an ID in the filter call. Otherwise the implicit OR default applies. Examples of using Multilink expressions would be - '1', '2', '-4', '3', '4', '-4', '-3' would search for IDs (1 or 2) and (3 or 4) - '-1' '-2' would search for all non-empty Multilinks filter_sql **Only in SQL backends** Lists the items that match the SQL provided. The SQL is a complete "select" statement. The SQL select must include the item id as the first column. This function **does not** filter out retired items, add on a where clause "__retired__ <> 1" if you don't want retired nodes. classhelp display a link to a javascript popup containing this class' "help" template. This generates a link to a popup window which displays the properties indicated by "properties" of the class named by "classname". The "properties" should be a comma-separated list (eg. 'id,name,description'). Properties defaults to all the properties of a class (excluding id, creator, created and activity). You may optionally override the "label" displayed, the "width", the "height", the number of items per page ("pagesize") and the field on which the list is sorted ("sort"). With the "filter" arg it is possible to specify a filter for which items are supposed to be displayed. It has to be of the format "=;=;...". The popup window will be resizable and scrollable. If the "property" arg is given, it's passed through to the javascript help_window function. This allows updating of a property in the calling HTML page. If the "form" arg is given, it's passed through to the javascript help_window function - it's the name of the form the "property" belongs to. submit generate a submit button (and action and @csrf hidden elements) renderWith render this class with the given template. history returns 'New node - no history' :) is_edit_ok is the user allowed to Edit the current class? is_view_ok is the user allowed to View the current class? =========== ============================================================= Note that if you have a property of the same name as one of the above methods, you'll need to access it using a python "item access" expression. For example:: python:context['list'] will access the "list" property, rather than the list method. Hyperdb item wrapper :::::::::::::::::::: This is implemented by the ``roundup.cgi.templating.HTMLItem`` class. This wrapper object provides access to a hyperdb item. We allow access to properties. There will be no "id" property. The value accessed through the property will be the current value of the same name from the CGI form. There are several methods available on these wrapper objects: =============== ======================================================== Method Description =============== ======================================================== submit generate a submit button (and action and @csrf hidden elements) journal return the journal of the current item (**not implemented**) history render the journal of the current item as HTML. By default properties marked as "quiet" (see `design documentation`_) are not shown unless the function is called with the showall=True parameter. Properties that are not Viewable to the user are not shown. renderQueryForm specific to the "query" class - render the search form for the query hasPermission specific to the "user" class - determine whether the user has a Permission. The signature is:: hasPermission(self, permission, [classname=], [property=], [itemid=]) where the classname defaults to the current context. hasRole specific to the "user" class - determine whether the user has a Role. The signature is:: hasRole(self, rolename) is_edit_ok is the user allowed to Edit the current item? is_view_ok is the user allowed to View the current item? is_retired is the item retired? download_url generate a url-quoted link for download of FileClass item contents (ie. file/) copy_url generate a url-quoted link for creating a copy of this item. By default, the copy will acquire all properties of the current item except for ``messages`` and ``files``. This can be overridden by passing ``exclude`` argument which contains a list (or any iterable) of property names that shall not be copied. Database-driven properties like ``id`` or ``activity`` cannot be copied. =============== ======================================================== Note that if you have a property of the same name as one of the above methods, you'll need to access it using a python "item access" expression. For example:: python:context['journal'] will access the "journal" property, rather than the journal method. Hyperdb property wrapper :::::::::::::::::::::::: This is implemented by subclasses of the ``roundup.cgi.templating.HTMLProperty`` class (``HTMLStringProperty``, ``HTMLNumberProperty``, and so on). This wrapper object provides access to a single property of a class. Its value may be either: 1. if accessed through a `hyperdb item wrapper`_, then it's a value from the hyperdb 2. if access through a `hyperdb class wrapper`_, then it's a value from the CGI form The property wrapper has some useful attributes: =============== ======================================================== Attribute Description =============== ======================================================== _name the name of the property _value the value of the property if any - this is the actual value retrieved from the hyperdb for this property =============== ======================================================== There are several methods available on these wrapper objects: =========== ================================================================ Method Description =========== ================================================================ plain render a "plain" representation of the property. This method may take two arguments: escape If true, escape the text so it is HTML safe (default: no). The reason this defaults to off is that text is usually escaped at a later stage by the TAL commands, unless the "structure" option is used in the template. The following ``tal:content`` expressions are all equivalent:: "structure python:msg.content.plain(escape=1)" "python:msg.content.plain()" "msg/content/plain" "msg/content" Usually you'll only want to use the escape option in a complex expression. hyperlink If true, turn URLs, email addresses and hyperdb item designators in the text into hyperlinks (default: no). Note that you'll need to use the "structure" TAL option if you want to use this ``tal:content`` expression:: "structure python:msg.content.plain(hyperlink=1)" The text is automatically HTML-escaped before the hyperlinking transformation done in the plain() method. hyperlinked The same as msg.content.plain(hyperlink=1), but nicer:: "structure msg/content/hyperlinked" field render an appropriate form edit field for the property - for most types this is a text entry box, but for Booleans it's a tri-state yes/no/neither selection. This method may take some arguments: size Sets the width in characters of the edit field format (Date properties only) Sets the format of the date in the field - uses the same format string argument as supplied to the ``pretty`` method below. popcal (Date properties only) Include the Javascript-based popup calendar for date selection. Defaults to on. stext only on String properties - render the value of the property as StructuredText (requires the StructureText module to be installed separately) multiline only on String properties - render a multiline form edit field for the property email only on String properties - render the value of the property as an obscured email address url_quote only on String properties. It quotes any characters in the string so it is safe to use in a url. E.G. a space is replaced with %20. confirm only on Password properties - render a second form edit field for the property, used for confirmation that the user typed the password correctly. Generates a field with name "name:confirm". now only on Date properties - return the current date as a new property reldate only on Date properties - render the interval between the date and now local only on Date properties - return this date as a new property with some timezone offset, for example:: python:context.creation.local(10) will render the date with a +10 hour offset. pretty Date properties - render the date as "dd Mon YYYY" (eg. "19 Mar 2004"). Takes an optional format argument, for example:: python:context.activity.pretty('%Y-%m-%d') Will format as "2004-03-19" instead. Interval properties - render the interval in a pretty format (eg. "yesterday"). The format arguments are those used in the standard ``strftime`` call (see the `Python Library Reference: time module`__) Number properties - takes a printf style format argument (default: '%0.3f') and formats the number accordingly. If the value can't be converted, '' is returned if the value is ``None`` otherwise it is converted to a string. popcal Generate a link to a popup calendar which may be used to edit the date field, for example:: you still need to include the ``field`` for the property, so typically you'd have:: menu only on Link and Multilink properties - render a form select list for this property. Takes a number of optional arguments size is used to limit the length of the list labels height is used to set the File   submit button will go here When a change is submitted, the system automatically generates a message describing the changed properties. As shown in the example, the editor template can use the ":note" and ":file" fields, which are added to the standard changenote message generated by Roundup. Form values ::::::::::: We have a number of ways to pull properties out of the form in order to meet the various needs of: 1. editing the current item (perhaps an issue item) 2. editing information related to the current item (eg. messages or attached files) 3. creating new information to be linked to the current item (eg. time spent on an issue) In the following, ```` values are variable, ":" may be one of ":" or "@", and other text ("required") is fixed. Properties are specified as form variables: ```` property on the current context item ``:`` property on the indicated item (for editing related information) ``-:`` property on the Nth new item of classname (generally for creating new items to attach to the current item) Once we have determined the "propname", we check to see if it is one of the special form values: ``@required`` The named property values must be supplied or a ValueError will be raised. ``@remove@=id(s)`` The ids will be removed from the multilink property. ``:add:=id(s)`` The ids will be added to the multilink property. ``:link:=`` Used to add a link to new items created during edit. These are collected and returned in ``all_links``. This will result in an additional linking operation (either Link set or Multilink append) after the edit/create is done using ``all_props`` in ``_editnodes``. The on the current item will be set/appended the id of the newly created item of class (where must be -). Any of the form variables may be prefixed with a classname or designator. Two special form values are supported for backwards compatibility: ``:note`` create a message (with content, author and date), linked to the context item. This is ALWAYS designated "msg-1". ``:file`` create a file, attached to the current item and any message created by :note. This is ALWAYS designated "file-1". Spool Section ~~~~~~~~~~~~~ The spool section lists related information like the messages and files of an issue. TODO History Section ~~~~~~~~~~~~~~~ The final section displayed is the history of the item - its database journal. This is generally generated with the template:: or:: if you want to show history entries for quiet properties. *To be done:* *The actual history entries of the item may be accessed for manual templating through the "journal" method of the item*:: a journal entry *where each journal entry is an HTMLJournalEntry.* Defining new web actions ------------------------ You may define new actions to be triggered by the ``@action`` form variable. These are added to the tracker ``extensions`` directory and registered using ``instance.registerAction``. All the existing Actions are defined in ``roundup.cgi.actions``. Adding action classes takes three steps; first you `define the new action class`_, then you `register the action class`_ with the cgi interface so it may be triggered by the ``@action`` form variable. Finally you `use the new action`_ in your HTML form. See "`setting up a "wizard" (or "druid") for controlled adding of issues`_" for an example. Define the new action class ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a new action class in your tracker's ``extensions`` directory, for example ``myaction.py``:: from roundup.cgi.actions import Action class MyAction(Action): def handle(self): ''' Perform some action. No return value is required. ''' The *self.client* attribute is an instance of ``roundup.cgi.client.Client``. See the docstring of that class for details of what it can do. The method will typically check the ``self.form`` variable's contents. It may then: - add information to ``self.client._ok_message`` or ``self.client._error_message`` (by using ``self.client.add_ok_message`` or ``self.client.add_error_message``, respectively) - change the ``self.client.template`` variable to alter what the user will see next - raise Unauthorised, SendStaticFile, SendFile, NotFound or Redirect exceptions (import them from roundup.cgi.exceptions) Register the action class ~~~~~~~~~~~~~~~~~~~~~~~~~~ The class is now written, but isn't available to the user until you register it with the following code appended to your ``myaction.py`` file:: def init(instance): instance.registerAction('myaction', myActionClass) This maps the action name "myaction" to the action class we defined. Use the new action ~~~~~~~~~~~~~~~~~~ In your HTML form, add a hidden form element like so:: where "myaction" is the name you registered in the previous step. Actions may return content to the user ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Actions generally perform some database manipulation and then pass control on to the rendering of a template in the current context (see `Determining web context`_ for how that works.) Some actions will want to generate the actual content returned to the user. Action methods may return their own content string to be displayed to the user, overriding the templating step. In this situation, we assume that the content is HTML by default. You may override the content type indicated to the user by calling ``setHeader``:: self.client.setHeader('Content-Type', 'text/csv') This example indicates that the value sent back to the user is actually comma-separated value content (eg. something to be loaded into a spreadsheet or database). 8-bit character set support in Web interface -------------------------------------------- The web interface uses UTF-8 default. It may be overridden in both forms and a browser cookie. - In forms, use the ``@charset`` variable. - To use the cookie override, have the ``roundup_charset`` cookie set. In both cases, the value is a valid charset name (eg. ``utf-8`` or ``kio8-r``). Inside Roundup, all strings are stored and processed in utf-8. Unfortunately, some older browsers do not work properly with utf-8-encoded pages (e.g. Netscape Navigator 4 displays wrong characters in form fields). This version allows one to change the character set for http transfers. To do so, you may add the following code to your ``page.html`` template:: utf-8 koi8-r (substitute ``koi8-r`` with appropriate charset for your language). Charset preference is kept in the browser cookie ``roundup_charset``. ``meta http-equiv`` lines added to the tracker templates in version 0.6.0 should be changed to include actual character set name:: The charset is also sent in the http header. .. _CustomExamples: Examples ======== .. contents:: :local: :depth: 2 Changing what's stored in the database -------------------------------------- The following examples illustrate ways to change the information stored in the database. Adding a new field to the classic schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This example shows how to add a simple field (a due date) to the default classic schema. It does not add any additional behaviour, such as enforcing the due date, or causing automatic actions to fire if the due date passes. You add new fields by editing the ``schema.py`` file in you tracker's home. Schema changes are automatically applied to the database on the next tracker access (note that roundup-server would need to be restarted as it caches the schema). .. index:: schema; example changes 1. Modify the ``schema.py``:: issue = IssueClass(db, "issue", assignedto=Link("user"), keyword=Multilink("keyword"), priority=Link("priority"), status=Link("status"), due_date=Date()) 2. Add an edit field to the ``issue.item.html`` template:: Due Date If you want to show only the date part of due_date then do this instead:: Due Date 3. Add the property to the ``issue.index.html`` page:: (in the heading row) Due Date (in the data row) If you want format control of the display of the due date you can enter the following in the data row to show only the actual due date::   4. Add the property to the ``issue.search.html`` page:: Due Date: 5. If you wish for the due date to appear in the standard views listed in the sidebar of the web interface then you'll need to add "due_date" to the columns and columns_showall lists in your ``page.html``:: columns string:id,activity,due_date,title,creator,status; columns_showall string:id,activity,due_date,title,creator,assignedto,status; Adding a new constrained field to the classic schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This example shows how to add a new constrained property (i.e. a selection of distinct values) to your tracker. Introduction :::::::::::: To make the classic schema of Roundup useful as a TODO tracking system for a group of systems administrators, it needs an extra data field per issue: a category. This would let sysadmins quickly list all TODOs in their particular area of interest without having to do complex queries, and without relying on the spelling capabilities of other sysadmins (a losing proposition at best). Adding a field to the database :::::::::::::::::::::::::::::: This is the easiest part of the change. The category would just be a plain string, nothing fancy. To change what is in the database you need to add some lines to the ``schema.py`` file of your tracker instance. Under the comment:: # add any additional database schema configuration here add:: category = Class(db, "category", name=String()) category.setkey("name") Here we are setting up a chunk of the database which we are calling "category". It contains a string, which we are refering to as "name" for lack of a more imaginative title. (Since "name" is one of the properties that Roundup looks for on items if you do not set a key for them, it's probably a good idea to stick with it for new classes if at all appropriate.) Then we are setting the key of this chunk of the database to be that "name". This is equivalent to an index for database types. This also means that there can only be one category with a given name. Adding the above lines allows us to create categories, but they're not tied to the issues that we are going to be creating. It's just a list of categories off on its own, which isn't much use. We need to link it in with the issues. To do that, find the lines in ``schema.py`` which set up the "issue" class, and then add a link to the category:: issue = IssueClass(db, "issue", ... , category=Multilink("category"), ... ) The ``Multilink()`` means that each issue can have many categories. If you were adding something with a one-to-one relationship to issues (such as the "assignedto" property), use ``Link()`` instead. That is all you need to do to change the schema. The rest of the effort is fiddling around so you can actually use the new category. Populating the new category class ::::::::::::::::::::::::::::::::: If you haven't initialised the database with the "``roundup-admin initialise``" command, then you can add the following to the tracker ``initial_data.py`` under the comment:: # add any additional database creation steps here - but only if you # haven't initialised the database with the admin "initialise" command Add:: category = db.getclass('category') category.create(name="scipy") category.create(name="chaco") category.create(name="weave") .. index:: roundup-admin; create entries in class If the database has already been initalised, then you need to use the ``roundup-admin`` tool:: % roundup-admin -i Roundup ready for input. Type "help" for help. roundup> create category name=scipy 1 roundup> create category name=chaco 2 roundup> create category name=weave 3 roundup> exit... There are unsaved changes. Commit them (y/N)? y Setting up security on the new objects :::::::::::::::::::::::::::::::::::::: By default only the admin user can look at and change objects. This doesn't suit us, as we want any user to be able to create new categories as required, and obviously everyone needs to be able to view the categories of issues for it to be useful. We therefore need to change the security of the category objects. This is also done in ``schema.py``. There are currently two loops which set up permissions and then assign them to various roles. Simply add the new "category" to both lists:: # Assign the access and edit permissions for issue, file and message # to regular users now for cl in 'issue', 'file', 'msg', 'category': p = db.security.getPermission('View', cl) db.security.addPermissionToRole('User', 'View', cl) db.security.addPermissionToRole('User', 'Edit', cl) db.security.addPermissionToRole('User', 'Create', cl) These lines assign the "View" and "Edit" Permissions to the "User" role, so that normal users can view and edit "category" objects. This is all the work that needs to be done for the database. It will store categories, and let users view and edit them. Now on to the interface stuff. Changing the web left hand frame :::::::::::::::::::::::::::::::: We need to give the users the ability to create new categories, and the place to put the link to this functionality is in the left hand function bar, under the "Issues" area. The file that defines how this area looks is ``html/page.html``, which is what we are going to be editing next. If you look at this file you can see that it contains a lot of "classblock" sections which are chunks of HTML that will be included or excluded in the output depending on whether the condition in the classblock is met. We are going to add the category code at the end of the classblock for the *issue* class::

Categories
New Category

The first two lines is the classblock definition, which sets up a condition that only users who have "View" permission for the "category" object will have this section included in their output. Next comes a plain "Categories" header in bold. Everyone who can view categories will get that. Next comes the link to the editing area of categories. This link will only appear if the condition - that the user has "Edit" permissions for the "category" objects - is matched. If they do have permission then they will get a link to another page which will let the user add new categories. Note that if you have permission to *view* but not to *edit* categories, then all you will see is a "Categories" header with nothing underneath it. This is obviously not very good interface design, but will do for now. I just claim that it is so I can add more links in this section later on. However, to fix the problem you could change the condition in the classblock statement, so that only users with "Edit" permission would see the "Categories" stuff. Setting up a page to edit categories :::::::::::::::::::::::::::::::::::: We defined code in the previous section which let users with the appropriate permissions see a link to a page which would let them edit conditions. Now we have to write that page. The link was for the *item* template of the *category* object. This translates into Roundup looking for a file called ``category.item.html`` in the ``html`` tracker directory. This is the file that we are going to write now. First, we add an info tag in a comment which doesn't affect the outcome of the code at all, but is useful for debugging. If you load a page in a browser and look at the page source, you can see which sections come from which files by looking for these comments:: Next we need to add in the METAL macro stuff so we get the normal page trappings:: Category editing

Category editing

Next we need to setup up a standard HTML form, which is the whole purpose of this file. We link to some handy javascript which sends the form through only once. This is to stop users hitting the send button multiple times when they are impatient and thus having the form sent multiple times::
Next we define some code which sets up the minimum list of fields that we require the user to enter. There will be only one field - "name" - so they better put something in it, otherwise the whole form is pointless:: To get everything to line up properly we will put everything in a table, and put a nice big header on it so the user has an idea what is happening:: Next, we need the field into which the user is going to enter the new category. The ``context.name.field(size=60)`` bit tells Roundup to generate a normal HTML field of size 60, and the contents of that field will be the "name" variable of the current context (namely "category"). The upshot of this is that when the user types something in to the form, a new category will be created with that name:: Then a submit button so that the user can submit the new category:: The ``context/submit`` bit generates the submit button but also generates the @action and @csrf hidden fields. The @action field is used to tell Roundup how to process the form. The @csrf field provides a unique single use token to defend against CSRF attacks. (More about anti-csrf measures can be found in ``upgrading.txt``.) Finally we finish off the tags we used at the start to do the METAL stuff:: So putting it all together, and closing the table and form we get:: Category editing This is quite a lot to just ask the user one simple question, but there is a lot of setup for basically one line (the form line) to do its work. To add another field to "category" would involve one more line (well, maybe a few extra to get the formatting correct). Adding the category to the issue :::::::::::::::::::::::::::::::: We now have the ability to create issues to our heart's content, but that is pointless unless we can assign categories to issues. Just like the ``html/category.item.html`` file was used to define how to add a new category, the ``html/issue.item.html`` is used to define how a new issue is created. Just like ``category.issue.html``, this file defines a form which has a table to lay things out. It doesn't matter where in the table we add new stuff, it is entirely up to your sense of aesthetics:: First, we define a nice header so that the user knows what the next section is, then the middle line does what we are most interested in. This ``context/category/field`` gets replaced by a field which contains the category in the current context (the current context being the new issue). The classhelp lines generate a link (labelled "list") to a popup window which contains the list of currently known categories. Searching on categories ::::::::::::::::::::::: Now we can add categories, and create issues with categories. The next obvious thing that we would like to be able to do, would be to search for issues based on their category, so that, for example, anyone working on the web server could look at all issues in the category "Web". If you look for "Search Issues" in the ``html/page.html`` file, you will find that it looks something like ``Search Issues``. This shows us that when you click on "Search Issues" it will be looking for a ``issue.search.html`` file to display. So that is the file that we will change. If you look at this file it should begin to seem familiar, although it does use some new macros. You can add the new category search code anywhere you like within that form:: The definitions in the ```` opening tag are used by the macros: - ``search_select`` expands to a drop-down box with all categories using ``db_klass`` and ``db_content``. - ``column_input`` expands to a checkbox for selecting what columns should be displayed. - ``sort_input`` expands to a radio button for selecting what property should be sorted on. - ``group_input`` expands to a radio button for selecting what property should be grouped on. The category search code above would expand to the following:: Adding category to the default view ::::::::::::::::::::::::::::::::::: We can now add categories, add issues with categories, and search for issues based on categories. This is everything that we need to do; however, there is some more icing that we would like. I think the category of an issue is important enough that it should be displayed by default when listing all the issues. Unfortunately, this is a bit less obvious than the previous steps. The code defining how the issues look is in ``html/issue.index.html``. This is a large table with a form down at the bottom for redisplaying and so forth. Firstly we need to add an appropriate header to the start of the table:: The *condition* part of this statement is to avoid displaying the Category column if the user has selected not to see it. The rest of the table is a loop which will go through every issue that matches the display criteria. The loop variable is "i" - which means that every issue gets assigned to "i" in turn. The new part of code to display the category will look like this:: The condition is the same as above: only display the condition when the user hasn't asked for it to be hidden. The next part is to set the content of the cell to be the category part of "i" - the current issue. Finally we have to edit ``html/page.html`` again. This time, we need to tell it that when the user clicks on "Unassigned Issues" or "All Issues", the category column should be included in the resulting list. If you scroll down the page file, you can see the links with lots of options. The option that we are interested in is the ``:columns=`` one which tells Roundup which fields of the issue to display. Simply add "category" to that list and it all should work. Adding a time log to your issues ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We want to log the dates and amount of time spent working on issues, and be able to give a summary of the total time spent on a particular issue. 1. Add a new class to your tracker ``schema.py``:: # storage for time logging timelog = Class(db, "timelog", period=Interval()) Note that we automatically get the date of the time log entry creation through the standard property "creation". You will need to grant "Creation" permission to the users who are allowed to add timelog entries. You may do this with:: db.security.addPermissionToRole('User', 'Create', 'timelog') db.security.addPermissionToRole('User', 'View', 'timelog') If users are also able to *edit* timelog entries, then also include:: db.security.addPermissionToRole('User', 'Edit', 'timelog') .. index:: schema; example changes 2. Link to the new class from your issue class (again, in ``schema.py``):: issue = IssueClass(db, "issue", assignedto=Link("user"), keyword=Multilink("keyword"), priority=Link("priority"), status=Link("status"), times=Multilink("timelog")) the "times" property is the new link to the "timelog" class. 3. We'll need to let people add in times to the issue, so in the web interface we'll have a new entry field. This is a special field because unlike the other fields in the ``issue.item`` template, it affects a different item (a timelog item) and not the template's item (an issue). We have a special syntax for form fields that affect items other than the template default item (see the cgi documentation on `special form variables`_). In particular, we add a field to capture a new timelog item's period:: and another hidden field that links that new timelog item (new because it's marked as having id "-1") to the issue item. It looks like this:: On submission, the "-1" timelog item will be created and assigned a real item id. The "times" property of the issue will have the new id added to it. The full entry will now look like this:: 4. We want to display a total of the timelog times that have been accumulated for an issue. To do this, we'll need to actually write some Python code, since it's beyond the scope of PageTemplates to perform such calculations. We do this by adding a module ``timespent.py`` to the ``extensions`` directory in our tracker. The contents of this file is as follows:: from roundup import date def totalTimeSpent(times): ''' Call me with a list of timelog items (which have an Interval "period" property) ''' total = date.Interval('0d') for time in times: total += time.period._value return total def init(instance): instance.registerUtil('totalTimeSpent', totalTimeSpent) We will now be able to access the ``totalTimeSpent`` function via the ``utils`` variable in our templates, as shown in the next step. 5. Display the timelog for an issue::
Category
Name name
  submit button will go here

Category editing
Category
Name name
  submit button will go here
Category
Priority:
Category:
Category
Time Log (enter as '3y 1m 4d 2:40:02' or parts thereof)
Time Log (enter as '3y 1m 4d 2:40:02' or parts thereof)
Time Log
DatePeriodLogged By
I put this just above the Messages log in my issue display. Note our use of the ``totalTimeSpent`` method which will total up the times for the issue and return a new Interval. That will be automatically displayed in the template as text like "+ 1y 2:40" (1 year, 2 hours and 40 minutes). 6. If you're using a persistent web server - ``roundup-server`` or ``mod_wsgi`` for example - then you'll need to restart that to pick up the code changes. When that's done, you'll be able to use the new time logging interface. An extension of this modification attaches the timelog entries to any change message entered at the time of the timelog entry: A. Add a link to the timelog to the msg class in ``schema.py``: msg = FileClass(db, "msg", author=Link("user", do_journal='no'), recipients=Multilink("user", do_journal='no'), date=Date(), summary=String(), files=Multilink("file"), messageid=String(), inreplyto=String(), times=Multilink("timelog")) B. Add a new hidden field that links that new timelog item (new because it's marked as having id "-1") to the new message. The link is placed in ``issue.item.html`` in the same section that handles the timelog entry. It looks like this after this addition:: Time Log (enter as '3y 1m 4d 2:40:02' or parts thereof) The "times" property of the message will have the new id added to it. C. Add the timelog listing from step 5. to the ``msg.item.html`` template so that the timelog entry appears on the message view page. Note that the call to totalTimeSpent is not used here since there will only be one single timelog entry for each message. I placed it after the Date entry like this:: Date:
Time Log
DatePeriodLogged By
Tracking different types of issues ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sometimes you will want to track different types of issues - developer, customer support, systems, sales leads, etc. A single Roundup tracker is able to support multiple types of issues. This example demonstrates adding a system support issue class to a tracker. 1. Figure out what information you're going to want to capture. OK, so this is obvious, but sometimes it's better to actually sit down for a while and think about the schema you're going to implement. 2. Add the new issue class to your tracker's ``schema.py``. Just after the "issue" class definition, add:: # list our systems system = Class(db, "system", name=String(), order=Number()) system.setkey("name") # store issues related to those systems support = IssueClass(db, "support", assignedto=Link("user"), keyword=Multilink("keyword"), status=Link("status"), deadline=Date(), affects=Multilink("system")) 3. Copy the existing ``issue.*`` (item, search and index) templates in the tracker's ``html`` to ``support.*``. Edit them so they use the properties defined in the ``support`` class. Be sure to check for hidden form variables like "required" to make sure they have the correct set of required properties. 4. Edit the modules in the ``detectors``, adding lines to their ``init`` functions where appropriate. Look for ``audit`` and ``react`` registrations on the ``issue`` class, and duplicate them for ``support``. 5. Create a new sidebar box for the new support class. Duplicate the existing issues one, changing the ``issue`` class name to ``support``. 6. Re-start your tracker and start using the new ``support`` class. Optionally, you might want to restrict the users able to access this new class to just the users with a new "SysAdmin" Role. To do this, we add some security declarations:: db.security.addPermissionToRole('SysAdmin', 'View', 'support') db.security.addPermissionToRole('SysAdmin', 'Create', 'support') db.security.addPermissionToRole('SysAdmin', 'Edit', 'support') You would then (as an "admin" user) edit the details of the appropriate users, and add "SysAdmin" to their Roles list. Alternatively, you might want to change the Edit/View permissions granted for the ``issue`` class so that it's only available to users with the "System" or "Developer" Role, and then the new class you're adding is available to all with the "User" Role. .. _external-authentication: Using External User Databases ----------------------------- Using an external password validation source ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: You will need to either have an "admin" user in your external password source *or* have one of your regular users have the Admin Role assigned. If you need to assign the Role *after* making the changes below, you may use the ``roundup-admin`` program to edit a user's details. We have a centrally-managed password changing system for our users. This results in a UN*X passwd-style file that we use for verification of users. Entries in the file consist of ``name:password`` where the password is encrypted using the standard UN*X ``crypt()`` function (see the ``crypt`` module in your Python distribution). An example entry would be:: admin:aamrgyQfDFSHw Each user of Roundup must still have their information stored in the Roundup database - we just use the passwd file to check their password. To do this, we need to override the standard ``verifyPassword`` method defined in ``roundup.cgi.actions.LoginAction`` and register the new class. The following is added as ``externalpassword.py`` in the tracker ``extensions`` directory:: import os, crypt from roundup.cgi.actions import LoginAction class ExternalPasswordLoginAction(LoginAction): def verifyPassword(self, userid, password): '''Look through the file, line by line, looking for a name that matches. ''' # get the user's username username = self.db.user.get(userid, 'username') # the passwords are stored in the "passwd.txt" file in the # tracker home file = os.path.join(self.db.config.TRACKER_HOME, 'passwd.txt') # see if we can find a match for ent in [line.strip().split(':') for line in open(file).readlines()]: if ent[0] == username: return crypt.crypt(password, ent[1][:2]) == ent[1] # user doesn't exist in the file return 0 def init(instance): instance.registerAction('login', ExternalPasswordLoginAction) You should also remove the redundant password fields from the ``user.item`` template. Using a UN*X passwd file as the user database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On some systems the primary store of users is the UN*X passwd file. It holds information on users such as their username, real name, password and primary user group. Roundup can use this store as its primary source of user information, but it needs additional information too - email address(es), Roundup Roles, vacation flags, Roundup hyperdb item ids, etc. Also, "retired" users must still exist in the user database, unlike some passwd files in which the users are removed when they no longer have access to a system. To make use of the passwd file, we therefore synchronise between the two user stores. We also use the passwd file to validate the user logins, as described in the previous example, `using an external password validation source`_. We keep the user lists in sync using a fairly simple script that runs once a day, or several times an hour if more immediate access is needed. In short, it: 1. parses the passwd file, finding usernames, passwords and real names, 2. compares that list to the current Roundup user list: a. entries no longer in the passwd file are *retired* b. entries with mismatching real names are *updated* c. entries only exist in the passwd file are *created* 3. send an email to administrators to let them know what's been done. The retiring and updating are simple operations, requiring only a call to ``retire()`` or ``set()``. The creation operation requires more information though - the user's email address and their Roundup Roles. We're going to assume that the user's email address is the same as their login name, so we just append the domain name to that. The Roles are determined using the passwd group identifier - mapping their UN*X group to an appropriate set of Roles. The script to perform all this, broken up into its main components, is as follows. Firstly, we import the necessary modules and open the tracker we're to work on:: import sys, os, smtplib from roundup import instance, date # open the tracker tracker_home = sys.argv[1] tracker = instance.open(tracker_home) Next we read in the *passwd* file from the tracker home:: # read in the users from the "passwd.txt" file file = os.path.join(tracker_home, 'passwd.txt') users = [x.strip().split(':') for x in open(file).readlines()] Handle special users (those to ignore in the file, and those who don't appear in the file):: # users to not keep ever, pre-load with the users I know aren't # "real" users ignore = ['ekmmon', 'bfast', 'csrmail'] # users to keep - pre-load with the roundup-specific users keep = ['comment_pool', 'network_pool', 'admin', 'dev-team', 'cs_pool', 'anonymous', 'system_pool', 'automated'] Now we map the UN*X group numbers to the Roles that users should have:: roles = { '501': 'User,Tech', # tech '502': 'User', # finance '503': 'User,CSR', # customer service reps '504': 'User', # sales '505': 'User', # marketing } Now we do all the work. Note that the body of the script (where we have the tracker database open) is wrapped in a ``try`` / ``finally`` clause, so that we always close the database cleanly when we're finished. So, we now do all the work:: # open the database db = tracker.open('admin') try: # store away messages to send to the tracker admins msg = [] # loop over the users list read in from the passwd file for user,passw,uid,gid,real,home,shell in users: if user in ignore: # this user shouldn't appear in our tracker continue keep.append(user) try: # see if the user exists in the tracker uid = db.user.lookup(user) # yes, they do - now check the real name for correctness if real != db.user.get(uid, 'realname'): db.user.set(uid, realname=real) msg.append('FIX %s - %s'%(user, real)) except KeyError: # nope, the user doesn't exist db.user.create(username=user, realname=real, address='%s@ekit-inc.com'%user, roles=roles[gid]) msg.append('ADD %s - %s (%s)'%(user, real, roles[gid])) # now check that all the users in the tracker are also in our # "keep" list - retire those who aren't for uid in db.user.list(): user = db.user.get(uid, 'username') if user not in keep: db.user.retire(uid) msg.append('RET %s'%user) # if we did work, then send email to the tracker admins if msg: # create the email msg = '''Subject: %s user database maintenance %s '''%(db.config.TRACKER_NAME, '\n'.join(msg)) # send the email smtp = smtplib.SMTP(db.config.MAILHOST) addr = db.config.ADMIN_EMAIL smtp.sendmail(addr, addr, msg) # now we're done - commit the changes db.commit() finally: # always close the database cleanly db.close() And that's it! Using an LDAP database for user information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A script that reads users from an LDAP store using https://pypi.org/project/python-ldap/ and then compares the list to the users in the Roundup user database would be pretty easy to write. You'd then have it run once an hour / day (or on demand if you can work that into your LDAP store workflow). See the example `Using a UN*X passwd file as the user database`_ for more information about doing this. To authenticate off the LDAP store (rather than using the passwords in the Roundup user database) you'd use the same python-ldap module inside an extension to the cgi interface. You'd do this by overriding the method called ``verifyPassword`` on the ``LoginAction`` class in your tracker's ``extensions`` directory (see `using an external password validation source`_). The method is implemented by default as:: def verifyPassword(self, userid, password): ''' Verify the password that the user has supplied ''' stored = self.db.user.get(self.userid, 'password') if password == stored: return 1 if not password and not stored: return 1 return 0 So you could reimplement this as something like:: def verifyPassword(self, userid, password): ''' Verify the password that the user has supplied ''' # look up some unique LDAP information about the user username = self.db.user.get(self.userid, 'username') # now verify the password supplied against the LDAP store Changes to Tracker Behaviour ---------------------------- .. index:: single: auditors; how to register (example) single: reactors; how to register (example) Preventing SPAM ~~~~~~~~~~~~~~~ The following detector code may be installed in your tracker's ``detectors`` directory. It will block any messages being created that have HTML attachments (a very common vector for spam and phishing) and any messages that have more than 2 HTTP URLs in them. Just copy the following into ``detectors/anti_spam.py`` in your tracker:: from roundup.exceptions import Reject def reject_html(db, cl, nodeid, newvalues): if newvalues['type'] == 'text/html': raise Reject('not allowed') def reject_manylinks(db, cl, nodeid, newvalues): content = newvalues['content'] if content.count('http://') > 2: raise Reject('not allowed') def init(db): db.file.audit('create', reject_html) db.msg.audit('create', reject_manylinks) You may also wish to block image attachments if your tracker does not need that ability:: if newvalues['type'].startswith('image/'): raise Reject('not allowed') Stop "nosy" messages going to people on vacation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When users go on vacation and set up vacation email bouncing, you'll start to see a lot of messages come back through Roundup "Fred is on vacation". Not very useful, and relatively easy to stop. 1. add a "vacation" flag to your users:: user = Class(db, "user", username=String(), password=Password(), address=String(), realname=String(), phone=String(), organisation=String(), alternate_addresses=String(), roles=String(), queries=Multilink("query"), vacation=Boolean()) 2. So that users may edit the vacation flags, add something like the following to your ``user.item`` template:: 3. edit your detector ``nosyreactor.py`` so that the ``nosyreaction()`` consists of:: def nosyreaction(db, cl, nodeid, oldvalues): users = db.user messages = db.msg # send a copy of all new messages to the nosy list for msgid in determineNewMessages(cl, nodeid, oldvalues): try: # figure the recipient ids sendto = [] seen_message = {} recipients = messages.get(msgid, 'recipients') for recipid in messages.get(msgid, 'recipients'): seen_message[recipid] = 1 # figure the author's id, and indicate they've received # the message authid = messages.get(msgid, 'author') # possibly send the message to the author, as long as # they aren't anonymous if (db.config.MESSAGES_TO_AUTHOR == 'yes' and users.get(authid, 'username') != 'anonymous'): sendto.append(authid) seen_message[authid] = 1 # now figure the nosy people who weren't recipients nosy = cl.get(nodeid, 'nosy') for nosyid in nosy: # Don't send nosy mail to the anonymous user (that # user shouldn't appear in the nosy list, but just # in case they do...) if users.get(nosyid, 'username') == 'anonymous': continue # make sure they haven't seen the message already if nosyid not in seen_message: # send it to them sendto.append(nosyid) recipients.append(nosyid) # generate a change note if oldvalues: note = cl.generateChangeNote(nodeid, oldvalues) else: note = cl.generateCreateNote(nodeid) # we have new recipients if sendto: # filter out the people on vacation sendto = [i for i in sendto if not users.get(i, 'vacation', 0)] # map userids to addresses sendto = [users.get(i, 'address') for i in sendto] # update the message's recipients list messages.set(msgid, recipients=recipients) # send the message cl.send_message(nodeid, msgid, note, sendto) except roundupdb.MessageSendError as message: raise roundupdb.DetectorError(message) Note that this is the standard nosy reaction code, with the small addition of:: # filter out the people on vacation sendto = [i for i in sendto if not users.get(i, 'vacation', 0)] which filters out the users that have the vacation flag set to true. Adding in state transition control ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sometimes tracker admins want to control the states to which users may move issues. You can do this by following these steps: 1. make "status" a required variable. This is achieved by adding the following to the top of the form in the ``issue.item.html`` template:: This will force users to select a status. 2. add a Multilink property to the status class:: stat = Class(db, "status", ... , transitions=Multilink('status'), ...) and then edit the statuses already created, either: a. through the web using the class list -> status class editor, or b. using the ``roundup-admin`` "set" command. 3. add an auditor module ``checktransition.py`` in your tracker's ``detectors`` directory, for example:: def checktransition(db, cl, nodeid, newvalues): ''' Check that the desired transition is valid for the "status" property. ''' if 'status' not in newvalues: return current = cl.get(nodeid, 'status') new = newvalues['status'] if new == current: return ok = db.status.get(current, 'transitions') if new not in ok: raise ValueError('Status not allowed to move from "%s" to "%s"'%( db.status.get(current, 'name'), db.status.get(new, 'name'))) def init(db): db.issue.audit('set', checktransition) 4. in the ``issue.item.html`` template, change the status editing bit from:: to:: which displays only the allowed status to transition to. Blocking issues that depend on other issues ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We needed the ability to mark certain issues as "blockers" - that is, they can't be resolved until another issue (the blocker) they rely on is resolved. To achieve this: 1. Create a new property on the ``issue`` class: ``blockers=Multilink("issue")``. To do this, edit the definition of this class in your tracker's ``schema.py`` file. Change this:: issue = IssueClass(db, "issue", assignedto=Link("user"), keyword=Multilink("keyword"), priority=Link("priority"), status=Link("status")) to this, adding the blockers entry:: issue = IssueClass(db, "issue", blockers=Multilink("issue"), assignedto=Link("user"), keyword=Multilink("keyword"), priority=Link("priority"), status=Link("status")) 2. Add the new ``blockers`` property to the ``issue.item.html`` edit page, using something like:: You'll need to fiddle with your item page layout to find an appropriate place to put it - I'll leave that fun part up to you. Just make sure it appears in the first table, possibly somewhere near the "superseders" field. 3. Create a new detector module (see below) which enforces the rules: - issues may not be resolved if they have blockers - when a blocker is resolved, it's removed from issues it blocks The contents of the detector should be something like this:: def blockresolution(db, cl, nodeid, newvalues): ''' If the issue has blockers, don't allow it to be resolved. ''' if nodeid is None: blockers = [] else: blockers = cl.get(nodeid, 'blockers') blockers = newvalues.get('blockers', blockers) # don't do anything if there's no blockers or the status hasn't # changed if not blockers or 'status' not in newvalues: return # get the resolved state ID resolved_id = db.status.lookup('resolved') # format the info u = db.config.TRACKER_WEB s = ', '.join(['%s'%( u,id,id) for id in blockers]) if len(blockers) == 1: s = 'issue %s is'%s else: s = 'issues %s are'%s # ok, see if we're trying to resolve if newvalues['status'] == resolved_id: raise ValueError("This issue can't be resolved until %s resolved."%s) def resolveblockers(db, cl, nodeid, oldvalues): ''' When we resolve an issue that's a blocker, remove it from the blockers list of the issue(s) it blocks. ''' newstatus = cl.get(nodeid,'status') # no change? if oldvalues.get('status', None) == newstatus: return resolved_id = db.status.lookup('resolved') # interesting? if newstatus != resolved_id: return # yes - find all the blocked issues, if any, and remove me from # their blockers list issues = cl.find(blockers=nodeid) for issueid in issues: blockers = cl.get(issueid, 'blockers') if nodeid in blockers: blockers.remove(nodeid) cl.set(issueid, blockers=blockers) def init(db): # might, in an obscure situation, happen in a create db.issue.audit('create', blockresolution) db.issue.audit('set', blockresolution) # can only happen on a set db.issue.react('set', resolveblockers) Put the above code in a file called "blockers.py" in your tracker's "detectors" directory. 4. Finally, and this is an optional step, modify the tracker web page URLs so they filter out issues with any blockers. You do this by adding an additional filter on "blockers" for the value "-1". For example, the existing "Show All" link in the "page" template (in the tracker's "html" directory) looks like this:: Show All
modify it to add the "blockers" info to the URL (note, both the "@filter" *and* "blockers" values must be specified):: Show All
The above examples are line-wrapped on the trailing & and should be unwrapped. That's it. You should now be able to set blockers on your issues. Note that if you want to know whether an issue has any other issues dependent on it (i.e. it's in their blockers list) you can look at the journal history at the bottom of the issue page - look for a "link" event to another issue's "blockers" property. Add users to the nosy list based on the keyword ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Let's say we need the ability to automatically add users to the nosy list based on the occurance of a keyword. Every user should be allowed to edit their own list of keywords for which they want to be added to the nosy list. Below, we'll show that this change can be done with minimal understanding of the Roundup system, using only copy and paste. This requires three changes to the tracker: a change in the database to allow per-user recording of the lists of keywords for which he wants to be put on the nosy list, a change in the user view allowing them to edit this list of keywords, and addition of an auditor which updates the nosy list when a keyword is set. Adding the nosy keyword list :::::::::::::::::::::::::::: The change to make in the database, is that for any user there should be a list of keywords for which he wants to be put on the nosy list. Adding a ``Multilink`` of ``keyword`` seems to fullfill this. As such, all that has to be done is to add a new field to the definition of ``user`` within the file ``schema.py``. We will call this new field ``nosy_keywords``, and the updated definition of user will be:: user = Class(db, "user", username=String(), password=Password(), address=String(), realname=String(), phone=String(), organisation=String(), alternate_addresses=String(), queries=Multilink('query'), roles=String(), timezone=String(), nosy_keywords=Multilink('keyword')) Changing the user view to allow changing the nosy keyword list :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: We want any user to be able to change the list of keywords for which he will by default be added to the nosy list. We choose to add this to the user view, as is generated by the file ``html/user.item.html``. We can easily see that the keyword field in the issue view has very similar editing requirements as our nosy keywords, both being lists of keywords. As such, we look for Keywords in ``issue.item.html``, and extract the associated parts from there. We add this to ``user.item.html`` at the bottom of the list of viewed items (i.e. just below the 'Alternate E-mail addresses' in the classic template):: Addition of an auditor to update the nosy list :::::::::::::::::::::::::::::::::::::::::::::: The more difficult part is the logic to add the users to the nosy list when required. We choose to perform this action whenever the keywords on an item are set (this includes the creation of items). Here we choose to start out with a copy of the ``detectors/nosyreaction.py`` detector, which we copy to the file ``detectors/nosy_keyword_reaction.py``. This looks like a good start as it also adds users to the nosy list. A look through the code reveals that the ``nosyreaction`` function actually sends the e-mail. We don't need this. Therefore, we can change the ``init`` function to:: def init(db): db.issue.audit('create', update_kw_nosy) db.issue.audit('set', update_kw_nosy) After that, we rename the ``updatenosy`` function to ``update_kw_nosy``. The first two blocks of code in that function relate to setting ``current`` to a combination of the old and new nosy lists. This functionality is left in the new auditor. The following block of code, which handled adding the assignedto user(s) to the nosy list in ``updatenosy``, should be replaced by a block of code to add the interested users to the nosy list. We choose here to loop over all new keywords, than looping over all users, and assign the user to the nosy list when the keyword occurs in the user's ``nosy_keywords``. The next part in ``updatenosy`` -- adding the author and/or recipients of a message to the nosy list -- is obviously not relevant here and is thus deleted from the new auditor. The last part, copying the new nosy list to ``newvalues``, can stay as is. This results in the following function:: def update_kw_nosy(db, cl, nodeid, newvalues): '''Update the nosy list for changes to the keywords ''' # nodeid will be None if this is a new node current = {} if nodeid is None: ok = ('new', 'yes') else: ok = ('yes',) # old node, get the current values from the node if they haven't # changed if 'nosy' not in newvalues: nosy = cl.get(nodeid, 'nosy') for value in nosy: if value not in current: current[value] = 1 # if the nosy list changed in this transaction, init from the new value if 'nosy' in newvalues: nosy = newvalues.get('nosy', []) for value in nosy: if not db.hasnode('user', value): continue if value not in current: current[value] = 1 # add users with keyword in nosy_keywords to the nosy list if 'keyword' in newvalues and newvalues['keyword'] is not None: keyword_ids = newvalues['keyword'] for keyword in keyword_ids: # loop over all users, # and assign user to nosy when keyword in nosy_keywords for user_id in db.user.list(): nosy_kw = db.user.get(user_id, "nosy_keywords") found = 0 for kw in nosy_kw: if kw == keyword: found = 1 if found: current[user_id] = 1 # that's it, save off the new nosy list newvalues['nosy'] = list(current.keys()) These two function are the only ones needed in the file. TODO: update this example to use the ``find()`` Class method. Caveats ::::::: A few problems with the design here can be noted: Multiple additions When a user, after automatic selection, is manually removed from the nosy list, he is added to the nosy list again when the keyword list of the issue is updated. A better design might be to only check which keywords are new compared to the old list of keywords, and only add users when they have indicated interest on a new keyword. The code could also be changed to only trigger on the ``create()`` event, rather than also on the ``set()`` event, thus only setting the nosy list when the issue is created. Scalability In the auditor, there is a loop over all users. For a site with only few users this will pose no serious problem; however, with many users this will be a serious performance bottleneck. A way out would be to link from the keywords to the users who selected these keywords as nosy keywords. This will eliminate the loop over all users. See the ``rev_multilink`` attribute to make this easier. Restricting updates that arrive by email ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roundup supports multiple update methods: 1. command line 2. plain email 3. pgp signed email 4. web access in some cases you may need to prevent changes to properties by some of these methods. For example you can set up issues that are viewable only by people on the nosy list. So you must prevent unauthenticated changes to the nosy list. Since plain email can be easily forged, it does not provide sufficient authentication in this senario. To prevent this we can add a detector that audits the source of the transaction and rejects the update if it changes the nosy list. Create the detector (auditor) module and add it to the detectors directory of your tracker:: from roundup import roundupdb, hyperdb from roundup.mailgw import Unauthorized def restrict_nosy_changes(db, cl, nodeid, newvalues): '''Do not permit changes to nosy via email.''' if 'nosy' not in newvalues: # the nosy field has not changed so no need to check. return if db.tx_Source in ['web', 'rest', 'xmlrpc', 'email-sig-openpgp', 'cli' ]: # if the source of the transaction is from an authenticated # source or a privileged process allow the transaction. # Other possible sources: 'email' return # otherwise raise an error raise Unauthorized( \ 'Changes to nosy property not allowed via %s for this issue.'%\ tx_Source) def init(db): ''' Install restrict_nosy_changes to run after other auditors. Allow initial creation email to set nosy. So don't execute: db.issue.audit('create', requestedbyauditor) Set priority to 110 to run this auditor after other auditors that can cause nosy to change. ''' db.issue.audit('set', restrict_nosy_changes, 110) This detector (auditor) will prevent updates to the nosy field if it arrives by email. Since it runs after other auditors (due to the priority of 110), it will also prevent changes to the nosy field that are done by other auditors if triggered by an email. Note that db.tx_Source was not present in roundup versions before 1.4.22, so you must be running a newer version to use this detector. Read the CHANGES.txt document in the roundup source code for further details on tx_Source. Changes to Security and Permissions ----------------------------------- Restricting the list of users that are assignable to a task ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. In your tracker's ``schema.py``, create a new Role, say "Developer":: db.security.addRole(name='Developer', description='A developer') 2. Just after that, create a new Permission, say "Fixer", specific to "issue":: p = db.security.addPermission(name='Fixer', klass='issue', description='User is allowed to be assigned to fix issues') 3. Then assign the new Permission to your "Developer" Role:: db.security.addPermissionToRole('Developer', p) 4. In the issue item edit page (``html/issue.item.html`` in your tracker directory), use the new Permission in restricting the "assignedto" list:: For extra security, you may wish to setup an auditor to enforce the Permission requirement (install this as ``assignedtoFixer.py`` in your tracker ``detectors`` directory):: def assignedtoMustBeFixer(db, cl, nodeid, newvalues): ''' Ensure the assignedto value in newvalues is used with the Fixer Permission ''' if 'assignedto' not in newvalues: # don't care return # get the userid userid = newvalues['assignedto'] if not db.security.hasPermission('Fixer', userid, cl.classname): raise ValueError('You do not have permission to edit %s'%cl.classname) def init(db): db.issue.audit('set', assignedtoMustBeFixer) db.issue.audit('create', assignedtoMustBeFixer) So now, if an edit action attempts to set "assignedto" to a user that doesn't have the "Fixer" Permission, the error will be raised. Users may only edit their issues ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In this case, users registering themselves are granted Provisional access, meaning they have access to edit the issues they submit, but not others. We create a new Role called "Provisional User" which is granted to newly-registered users, and has limited access. One of the Permissions they have is the new "Edit Own" on issues (regular users have "Edit".) First up, we create the new Role and Permission structure in ``schema.py``:: # # New users not approved by the admin # db.security.addRole(name='Provisional User', description='New user registered via web or email') # These users need to be able to view and create issues but only edit # and view their own db.security.addPermissionToRole('Provisional User', 'Create', 'issue') def own_issue(db, userid, itemid): '''Determine whether the userid matches the creator of the issue.''' return userid == db.issue.get(itemid, 'creator') p = db.security.addPermission(name='Edit', klass='issue', check=own_issue, description='Can only edit own issues') db.security.addPermissionToRole('Provisional User', p) p = db.security.addPermission(name='View', klass='issue', check=own_issue, description='Can only view own issues') db.security.addPermissionToRole('Provisional User', p) # This allows the interface to get the names of the properties # in the issue. Used for selecting sorting and grouping # on the index page. p = db.security.addPermission(name='Search', klass='issue') db.security.addPermissionToRole ('Provisional User', p) # Assign the Permissions for issue-related classes for cl in 'file', 'msg', 'query', 'keyword': db.security.addPermissionToRole('Provisional User', 'View', cl) db.security.addPermissionToRole('Provisional User', 'Edit', cl) db.security.addPermissionToRole('Provisional User', 'Create', cl) for cl in 'priority', 'status': db.security.addPermissionToRole('Provisional User', 'View', cl) # and give the new users access to the web and email interface db.security.addPermissionToRole('Provisional User', 'Web Access') db.security.addPermissionToRole('Provisional User', 'Email Access') # make sure they can view & edit their own user record def own_record(db, userid, itemid): '''Determine whether the userid matches the item being accessed.''' return userid == itemid p = db.security.addPermission(name='View', klass='user', check=own_record, description="User is allowed to view their own user details") db.security.addPermissionToRole('Provisional User', p) p = db.security.addPermission(name='Edit', klass='user', check=own_record, description="User is allowed to edit their own user details") db.security.addPermissionToRole('Provisional User', p) Then, in ``config.ini``, we change the Role assigned to newly-registered users, replacing the existing ``'User'`` values:: [main] ... new_web_user_roles = Provisional User new_email_user_roles = Provisional User All users may only view and edit issues, files and messages they create ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Replace the standard "classic" tracker View and Edit Permission assignments for the "issue", "file" and "msg" classes with the following:: def checker(klass): def check(db, userid, itemid, klass=klass): return db.getclass(klass).get(itemid, 'creator') == userid return check for cl in 'issue', 'file', 'msg': p = db.security.addPermission(name='View', klass=cl, check=checker(cl), description='User can view only if creator.') db.security.addPermissionToRole('User', p) p = db.security.addPermission(name='Edit', klass=cl, check=checker(cl), description='User can edit only if creator.') db.security.addPermissionToRole('User', p) db.security.addPermissionToRole('User', 'Create', cl) # This allows the interface to get the names of the properties # in the issue. Used for selecting sorting and grouping # on the index page. p = db.security.addPermission(name='Search', klass='issue') db.security.addPermissionToRole ('User', p) Moderating user registration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You could set up new-user moderation in a public tracker by: 1. creating a new highly-restricted user role "Pending", 2. set the config new_web_user_roles and/or new_email_user_roles to that role, 3. have an auditor that emails you when new users are created with that role using roundup.mailer 4. edit the role to "User" for valid users. Some simple javascript might help in the last step. If you have high volume you could search for all currently-Pending users and do a bulk edit of all their roles at once (again probably with some simple javascript help). Changes to the Web User Interface --------------------------------- Adding action links to the index page ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Add a column to the ``item.index.html`` template. Resolving the issue:: resolve "Take" the issue:: take ... and so on. Colouring the rows in the issue index according to priority ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A simple ``tal:attributes`` statement will do the bulk of the work here. In the ``issue.index.html`` template, add this to the ```` that displays the rows of data:: and then in your stylesheet (``style.css``) specify the colouring for the different priorities, as follows:: tr.priority-critical td { background-color: red; } tr.priority-urgent td { background-color: orange; } and so on, with far less offensive colours :) Editing multiple items in an index view ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To edit the status of all items in the item index view, edit the ``issue.index.html``: 1. add a form around the listing table (separate from the existing index-page form), so at the top it reads::
On Vacation vacation
Status statusStatus Waiting On
View:
Nosy Keywords
and at the bottom of that table::
`` from the list table, not the navigation table or the subsequent form table. 2. in the display for the issue property, change::   to::   this will result in an edit field for the status property. 3. after the ``tal:block`` which lists the index items (marked by ``tal:repeat="i batch"``) add a new table row:: which gives us a submit button, indicates that we are performing an edit on any changed statuses, and provides a defense against cross site request forgery attacks. The final ``tal:block`` will make sure that the current index view parameters (filtering, columns, etc) will be used in rendering the next page (the results of the editing). Displaying only message summaries in the issue display ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Alter the ``issue.item`` template section for messages to::
Messages
author date summary remove
Enabling display of either message summaries or the entire messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is pretty simple - all we need to do is copy the code from the example `displaying only message summaries in the issue display`_ into our template alongside the summary display, and then introduce a switch that shows either the one or the other. We'll use a new form variable, ``@whole_messages`` to achieve this::
Messages show entire messages
author date summary remove
Messages show only summaries
author date (remove)
Setting up a "wizard" (or "druid") for controlled adding of issues ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Set up the page templates you wish to use for data input. My wizard is going to be a two-step process: first figuring out what category of issue the user is submitting, and then getting details specific to that category. The first page includes a table of help, explaining what the category names mean, and then the core of the form::
Category: The next page has the usual issue entry information, with the addition of the following form fragments::
. . .
Note that later in the form, I use the value of "cat" to decide which form elements should be displayed. For example:: Operating System Web Browser ... the above section will only be displayed if the category is one of 6, 10, 13, 14, 15, 16 or 17. 3. Determine what actions need to be taken between the pages - these are usually to validate user choices and determine what page is next. Now encode those actions in a new ``Action`` class (see `defining new web actions`_):: from roundup.cgi.actions import Action class Page1SubmitAction(Action): def handle(self): ''' Verify that the user has selected a category, and then move on to page 2. ''' category = self.form['category'].value if category == '-1': self.client.add_error_message('You must select a category of report') return # everything's ok, move on to the next page self.client.template = 'add_page2' def init(instance): instance.registerAction('page1_submit', Page1SubmitAction) 4. Use the usual "new" action as the ``@action`` on the final page, and you're done (the standard context/submit method can do this for you). Silent Submit ~~~~~~~~~~~~~ When working on an issue, most of the time the people on the nosy list need to be notified of changes. There are cases where a user wants to add a comment to an issue and not bother other users on the nosy list. This feature is called Silent Submit because it allows the user to silently modify an issue and not tell anyone. There are several parts to this change. The main activity part involves editing the stock detectors/nosyreaction.py file in your tracker. Insert the following lines near the top of the nosyreaction function:: # Did user click button to do a silent change? try: if db.web['submit'] == "silent_change": return except (AttributeError, KeyError) as err: # The web attribute or submit key don't exist. # That's fine. We were probably triggered by an email # or cli based change. pass This checks the submit button to see if it is the silent type. If there are exceptions trying to make that determination they are ignored and processing continues. You may wonder how db.web gets set. This is done by creating an extension. Add the file extensions/edit.py with this content:: from roundup.cgi.actions import EditItemAction class Edit2Action(EditItemAction): def handle(self): self.db.web = {} # create the dict # populate the dict by getting the value of the submit_button # element from the form. self.db.web['submit'] = self.form['submit_button'].value # call the core EditItemAction to process the edit. EditItemAction.handle(self) def init(instance): '''Override the default edit action with this new version''' instance.registerAction('edit', Edit2Action) This code is a wrapper for the Roundup EditItemAction. It checks the form's submit button to save the value element. The rest of the changes needed for the Silent Submit feature involves editing html/issue.item.html to add the silent submit button. In the stock issue.item.html the submit button is on a line that contains "submit button". Replace that line with something like the following::   Note the difference in the value attribute for the two submit buttons. The value "silent_change" in the button specification must match the string in the nosy reaction function. Debugging Trackers ================== There are three switches in tracker configs that turn on debugging in Roundup: 1. web :: debug 2. mail :: debug 3. logging :: level See the config.ini file or the `tracker configuration`_ section above for more information. Additionally, the ``roundup-server.py`` script has its own debugging mode in which it reloads edited templates immediately when they are changed, rather than requiring a web server restart. .. _`design documentation`: design.html .. _`developer's guide`: developers.html .. _`rest interface documentation`: rest.html#programming-the-rest-api .. _change the rate limiting method: rest.html#creating-custom-rate-limits .. _`directions in the rest interface documentation`: rest.html#enabling-the-rest-api .. _`xmlrpc interface documentation`: xmlrpc.html#through-roundup .. _`zxcvbn`: https://github.com/dwolfhub/zxcvbn-python code-1a221cf0758b/doc/debugging.txt0000644000000000000000000000132014427061170015100 0ustar 00000000000000Debugging Aids -------------- Try turning on logging of DEBUG level messages. This may be done a number of ways, depending on what it is you're testing: 1. If you're testing the database unit tests, then set the environment variable ``LOGGING_LEVEL=DEBUG``. This may be done like so: LOGGING_LEVEL=DEBUG python -m pytest test/ This variable replaces the older HYPERDBDEBUG environment var. 2. If you're testing a particular tracker, then set the logging level in your tracker's ``config.ini``. SENDMAILDEBUG ============= Set to a filename and roundup will write each email message that it sends to that file instead to the internet. This environment variable is independent of the python -O flag. code-1a221cf0758b/doc/default.css0000644000000000000000000000753114427061170014554 0ustar 00000000000000/* :Author: David Goodger :Contact: goodger@users.sourceforge.net :date: 2004-06-09 00:25:32 :copyright: This stylesheet has been placed in the public domain. Default cascading style sheet for the HTML output of Docutils. */ a.target { color: blue } a.toc-backref { text-decoration: none ; color: black } dd { margin-bottom: 0.5em } div.abstract { margin: 2em 5em } div.abstract p.topic-title { font-weight: bold ; text-align: center } div.attention, div.caution, div.danger, div.error, div.important, div.tip, div.warning { margin: 2em ; border: medium outset ; padding: 1em } div.hint, div.note { font-size: 80%; float: right; width: 15em; margin: 0.5em; margin-left: 1em ; border: solid #aaa; background: #eee; padding: 1em; } div.attention p.admonition-title, div.caution p.admonition-title, div.danger p.admonition-title, div.error p.admonition-title, div.warning p.admonition-title { color: red ; font-weight: bold ; font-family: sans-serif } div.hint p.admonition-title, div.important p.admonition-title, div.note p.admonition-title, div.tip p.admonition-title { font-weight: bold ; font-family: sans-serif } div.dedication { margin: 2em 5em ; text-align: center ; font-style: italic } div.dedication p.topic-title { font-weight: bold ; font-style: normal } div.figure { margin-left: 2em } div.footer, div.header { font-size: smaller } div.system-messages { margin: 5em } div.system-messages h1 { color: red } div.system-message { border: medium outset ; padding: 1em } div.system-message p.system-message-title { color: red ; font-weight: bold } div.topic { margin: 2em } h1 { margin-top: 2em; text-decoration: underline; } h1.title { text-align: center; margin-top: .5em; } h2.subtitle { text-align: center } hr { width: 75% } ol.simple, ul.simple { margin-top: 0; margin-bottom: 1em } ol.arabic { list-style: decimal } ol.loweralpha { list-style: lower-alpha } ol.upperalpha { list-style: upper-alpha } ol.lowerroman { list-style: lower-roman } ol.upperroman { list-style: upper-roman } p.caption { font-style: italic } p.credits { font-style: italic ; font-size: smaller } p.first { margin-top: 0 } p.label { white-space: nowrap } p.topic-title { font-weight: bold } pre.address { margin-bottom: 0 ; margin-top: 0 ; font-family: serif ; font-size: 100% } pre.line-block { font-family: serif ; font-size: 100% } pre.literal-block, pre.doctest-block { margin-left: 2em ; margin-right: 2em ; background-color: #eeeeee } span.classifier { font-family: sans-serif ; font-style: oblique } span.classifier-delimiter { font-family: sans-serif ; font-weight: bold } span.field-argument { font-style: italic } span.interpreted { font-family: sans-serif } span.option-argument { font-style: italic } span.pre { white-space: pre } span.problematic { color: red } table { margin-top: 0.5em ; margin-bottom: 0.5em ; } table.citation { border-top: 0; border-bottom: 0; border-right: 0; border-left: solid thin gray ; padding-left: 0.5ex } table.docinfo { margin: 2em 4em } table.footnote { border-left: solid thin black ; padding-left: 0.5ex } td, th { padding-left: 0.5em ; padding-right: 0.5em ; vertical-align: baseline; } table.table { border-spacing: 0px; border-collapse: separate; } table.table td { text-align: left; border: solid thin gray; } table.table th { text-align: left; border: solid thin gray; } td > p:first-child, th > p:first-child { margin-top: 0em } th.docinfo-name { font-weight: bold ; text-align: right } th.field-name { font-weight: bold ; text-align: right } h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt { font-size: 100% } tt { background-color: #eeeeee } tt.literal span.pre { background-color: #eeeeee } ul.auto-toc { list-style-type: none } code-1a221cf0758b/doc/design.txt0000644000000000000000000020273714427061170014435 0ustar 00000000000000.. meta:: :description: Documentation of design for the Roundup Issue Tracker. This is the original design (written by Ka-Ping Yee) updated with implementation notes by Richard Jones. ======================================================== Roundup - An Issue-Tracking System for Knowledge Workers ======================================================== :Authors: Ka-Ping Yee (original), Richard Jones (implementation) Contents .. contents:: :local: Introduction --------------- This document presents a description of the components of the Roundup system and specifies their interfaces and behaviour in sufficient detail to guide an implementation. For the philosophy and rationale behind the Roundup design, see the first-round Software Carpentry `submission for Roundup`__. This document fleshes out that design as well as specifying interfaces so that the components can be developed separately. __ spec.html The Layer Cake ----------------- Lots of software design documents come with a picture of a cake. Everybody seems to like them. I also like cakes (I think they are tasty). So I, too, shall include a picture of a cake here:: ________________________________________________________________ | E-mail Client | Web Browser | Detector Scripts | Shell | |---------------+---------------+--------------------+-----------| | E-mail User | Web User | Detector | Command | |----------------------------------------------------------------| | Roundup Database Layer | |----------------------------------------------------------------| | Hyperdatabase Layer | |----------------------------------------------------------------| | Storage Layer | ---------------------------------------------------------------- The colourful parts of the cake are part of our system; the faint grey parts of the cake are external components. I will now proceed to forgo all table manners and eat from the bottom of the cake to the top. You may want to stand back a bit so you don't get covered in crumbs. Hyperdatabase ------------- The lowest-level component to be implemented is the hyperdatabase. The hyperdatabase is a flexible data store that can hold configurable data in records which we call items. The hyperdatabase is implemented on top of the storage layer, an external module for storing its data. The "batteries-includes" distribution implements the hyperdatabase on the standard anydbm module. The storage layer could be a third-party RDBMS; for a low-maintenance solution, implementing the hyperdatabase on the SQLite RDBMS is suggested. Dates and Date Arithmetic ~~~~~~~~~~~~~~~~~~~~~~~~~ Before we get into the hyperdatabase itself, we need a way of handling dates. The hyperdatabase module provides Timestamp objects for representing date-and-time stamps and Interval objects for representing date-and-time intervals. As strings, date-and-time stamps are specified with the date in ISO8601 international standard format (``yyyy-mm-dd``) joined to the time (``hh:mm:ss``) by a period "``.``". Dates in this form can be easily compared and are fairly readable when printed. An example of a valid stamp is "``2000-06-24.13:03:59``". We'll call this the "full date format". When Timestamp objects are printed as strings, they appear in the full date format with the time always given in GMT. The full date format is always exactly 19 characters long. For user input, some partial forms are also permitted: the whole time or just the seconds may be omitted; and the whole date may be omitted or just the year may be omitted. If the time is given, the time is interpreted in the user's local time zone. The Date constructor takes care of these conversions. In the following examples, suppose that ``yyyy`` is the current year, ``mm`` is the current month, and ``dd`` is the current day of the month; and suppose that the user is on Eastern Standard Time. - "2000-04-17" means - "01-25" means - "2000-04-17.03:45" means - "08-13.22:13" means - "11-07.09:32:43" means - "14:25" means - "8:47:11" means - the special date "." means "right now" Date intervals are specified using the suffixes "y", "m", and "d". The suffix "w" (for "week") means 7 days. Time intervals are specified in hh:mm:ss format (the seconds may be omitted, but the hours and minutes may not). - "3y" means three years - "2y 1m" means two years and one month - "1m 25d" means one month and 25 days - "2w 3d" means two weeks and three days - "1d 2:50" means one day, two hours, and 50 minutes - "14:00" means 14 hours - "0:04:33" means four minutes and 33 seconds The Date class should understand simple date expressions of the form *stamp* ``+`` *interval* and *stamp* ``-`` *interval*. When adding or subtracting intervals involving months or years, the components are handled separately. For example, when evaluating "``2000-06-25 + 1m 10d``", we first add one month to get 2000-07-25, then add 10 days to get 2000-08-04 (rather than trying to decide whether 1m 10d means 38 or 40 or 41 days). Here is an outline of the Date and Interval classes:: class Date: def __init__(self, spec, offset): """Construct a date given a specification and a time zone offset. 'spec' is a full date or a partial form, with an optional added or subtracted interval. 'offset' is the local time zone offset from GMT in hours. """ def __add__(self, interval): """Add an interval to this date to produce another date.""" def __sub__(self, interval): """Subtract an interval from this date to produce another date. """ def __cmp__(self, other): """Compare this date to another date.""" def __str__(self): """Return this date as a string in the yyyy-mm-dd.hh:mm:ss format. """ def local(self, offset): """Return this date as yyyy-mm-dd.hh:mm:ss in a local time zone. """ class Interval: def __init__(self, spec): """Construct an interval given a specification.""" def __cmp__(self, other): """Compare this interval to another interval.""" def __str__(self): """Return this interval as a string.""" Here are some examples of how these classes would behave in practice. For the following examples, assume that we are on Eastern Standard Time and the current local time is 19:34:02 on 25 June 2000:: >>> Date(".") >>> _.local(-5) "2000-06-25.19:34:02" >>> Date(". + 2d") >>> Date("1997-04-17", -5) >>> Date("01-25", -5) >>> Date("08-13.22:13", -5) >>> Date("14:25", -5) >>> Interval(" 3w 1 d 2:00") >>> Date(". + 2d") - Interval("3w") Items and Classes ~~~~~~~~~~~~~~~~~ Items contain data in properties. To Python, these properties are presented as the key-value pairs of a dictionary. Each item belongs to a class which defines the names and types of its properties. The database permits the creation and modification of classes as well as items. Identifiers and Designators ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each item has a numeric identifier which is unique among items in its class. The items are numbered sequentially within each class in order of creation, starting from 1. The designator for an item is a way to identify an item in the database, and consists of the name of the item's class concatenated with the item's numeric identifier. For example, if "spam" and "eggs" are classes, the first item created in class "spam" has id 1 and designator "spam1". The first item created in class "eggs" also has id 1 but has the distinct designator "eggs1". Item designators are conventionally enclosed in square brackets when mentioned in plain text. This permits a casual mention of, say, "[patch37]" in an e-mail message to be turned into an active hyperlink. Property Names and Types ~~~~~~~~~~~~~~~~~~~~~~~~ Property names must begin with a letter. A property may be one of five basic types: - String properties are for storing arbitrary-length strings. - Boolean properties are for storing true/false, or yes/no values. - Integer properties are for storing Integer (non real) numeric values. - Number properties are for storing numeric values. - Date properties store date-and-time stamps. Their values are Timestamp objects. - A Link property refers to a single other item selected from a specified class. The class is part of the property; the value is an integer, the id of the chosen item. - A Multilink property refers to possibly many items in a specified class. The value is a list of integers. *None* is also a permitted value for any of these property types. An attempt to store None into a Multilink property stores an empty list. A property that is not specified will return as None from a *get* operation. Hyperdb Interface Specification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ TODO: replace the Interface Specifications with links to the pydoc The hyperdb module provides property objects to designate the different kinds of properties. All property objects support the following settings: quiet=False: if set to True, changes to the property will not be shown to the user. This can be used for administrative properties that are automatically updated when a user makes some other change. This reduces confusion by the user and clutter in the display. The property change will not be shown in: - the change confirmation message when a change is entered in the web interface - the property change section of the change note email ("nosy email") - the web history shown at the bottom of an item page required=False: if set to True, the property name is returned when calling get_required_props(self, propnames = []). Any additional props specified in propnames is merged with the required props. default_value=None or [] depending on object type: this sets the default value if none is specified. The default value can be retrieved by calling the get_default_value() method on the property object. E.G. assuming title is part of an Issue:: title=String(required=True, default_value="not set",quiet=True) will create a property called ``title`` that will be included in the get_required_props() output. Calling db.issue.properties['title'].get_default_value() will return "not set". Changes to the property will not be displayed in: - emailed change notes, - the history at the end of the item pages in the web interface - in the confirmation notice (displayed as a green banner) shown on changes. These objects are used when specifying what properties belong in classes:: class String: def __init__(self, indexme='no'): """An object designating a String property.""" class Boolean: def __init__(self): """An object designating a Boolean property.""" class Integer: def __init__(self): """An object designating an Integer property.""" class Number: def __init__(self): """An object designating a Number property.""" class Date: def __init__(self): """An object designating a Date property.""" class Link: def __init__(self, classname, do_journal='yes'): """An object designating a Link property that links to items in a specified class. If the do_journal argument is not 'yes' then changes to the property are not journalled in the linked item. """ class Multilink: def __init__(self, classname, do_journal='yes'): """An object designating a Multilink property that links to items in a specified class. If the do_journal argument is not 'yes' then changes to the property are not journalled in the linked item(s). """ Here is the interface provided by the hyperdatabase:: class Database: """A database for storing records containing flexible data types. """ def __init__(self, config, journaltag=None): """Open a hyperdatabase given a specifier to some storage. The 'storagelocator' is obtained from config.DATABASE. The meaning of 'storagelocator' depends on the particular implementation of the hyperdatabase. It could be a file name, a directory path, a socket descriptor for a connection to a database over the network, etc. The 'journaltag' is a token that will be attached to the journal entries for any edits done on the database. If 'journaltag' is None, the database is opened in read-only mode: the Class.create(), Class.set(), Class.retire(), and Class.restore() methods are disabled. """ def __getattr__(self, classname): """A convenient way of calling self.getclass(classname).""" def getclasses(self): """Return a list of the names of all existing classes.""" def getclass(self, classname): """Get the Class object representing a particular class. If 'classname' is not a valid class name, a KeyError is raised. """ class Class: """The handle to a particular class of items in a hyperdatabase. """ def __init__(self, db, classname, **properties): """Create a new class with a given name and property specification. 'classname' must not collide with the name of an existing class, or a ValueError is raised. 'classname' must start with an alphabetic letter. It must end with an alphabetic letter or '_'. Internal characters can be alphanumeric or '_'. ValueError is raised if the classname is not correct. The keyword arguments in 'properties' must map names to property objects, or a TypeError is raised. A proxied reference to the database is available as the 'db' attribute on instances. For example, in 'IssueClass.send_message', the following is used to lookup users, messages and files:: users = self.db.user messages = self.db.msg files = self.db.file """ # Editing items: def create(self, **propvalues): """Create a new item of this class and return its id. The keyword arguments in 'propvalues' map property names to values. The values of arguments must be acceptable for the types of their corresponding properties or a TypeError is raised. If this class has a key property, it must be present and its value must not collide with other key strings or a ValueError is raised. Any other properties on this class that are missing from the 'propvalues' dictionary are set to None. If an id in a link or multilink property does not refer to a valid item, an IndexError is raised. """ def get(self, itemid, propname): """Get the value of a property on an existing item of this class. 'itemid' must be the id of an existing item of this class or an IndexError is raised. 'propname' must be the name of a property of this class or a KeyError is raised. """ def set(self, itemid, **propvalues): """Modify a property on an existing item of this class. 'itemid' must be the id of an existing item of this class or an IndexError is raised. Each key in 'propvalues' must be the name of a property of this class or a KeyError is raised. All values in 'propvalues' must be acceptable types for their corresponding properties or a TypeError is raised. If the value of the key property is set, it must not collide with other key strings or a ValueError is raised. If the value of a Link or Multilink property contains an invalid item id, a ValueError is raised. """ def retire(self, itemid): """Retire an item. The properties on the item remain available from the get() method, and the item's id is never reused. Retired items are not returned by the find(), list(), or lookup() methods, and other items may reuse the values of their key properties. """ def restore(self, nodeid): '''Restore a retired node. Make node available for all operations like it was before retirement. ''' def history(self, itemid): """Retrieve the journal of edits on a particular item. 'itemid' must be the id of an existing item of this class or an IndexError is raised. The returned list contains tuples of the form (date, tag, action, params) 'date' is a Timestamp object specifying the time of the change and 'tag' is the journaltag specified when the database was opened. 'action' may be: 'create' or 'set' -- 'params' is a dictionary of property values 'link' or 'unlink' -- 'params' is (classname, itemid, propname) 'retire' -- 'params' is None """ # Locating items: def setkey(self, propname): """Select a String property of this class to be the key property. 'propname' must be the name of a String property of this class or None, or a TypeError is raised. The values of the key property on all existing items must be unique or a ValueError is raised. """ def getkey(self): """Return the name of the key property for this class or None. """ def lookup(self, keyvalue): """Locate a particular item by its key property and return its id. If this class has no key property, a TypeError is raised. If the 'keyvalue' matches one of the values for the key property among the items in this class, the matching item's id is returned; otherwise a KeyError is raised. """ def find(self, **propspec): """Get the ids of items in this class which link to the given items. 'propspec' consists of keyword args propname=itemid or propname={:1, :1, ...} 'propname' must be the name of a property in this class, or a KeyError is raised. That property must be a Link or Multilink property, or a TypeError is raised. Any item in this class whose 'propname' property links to any of the itemids will be returned. Examples:: db.issue.find(messages='1') db.issue.find(messages={'1':1,'3':1}, files={'7':1}) """ def filter(self, search_matches, filterspec, sort, group, retired, exact_match_spec, limit, offset): """Return a list of the ids of the active nodes in this class that match the 'filter' spec, sorted by the group spec and then the sort spec. The arguments sort, group, retired, and exact_match_spec are optional. "search_matches" is a container type which by default is None and optionally contains IDs of items to match. If non-empty only IDs of the initial set are returned. "filterspec" is {propname: value(s)} "exact_match_spec" is the same format as "filterspec" but specifies exact match for the given propnames. This only makes a difference for String properties, these specify case insensitive substring search when in "filterspec" and exact match when in exact_match_spec. "sort" and "group" are [(dir, prop), ...] where dir is '+', '-' or None and prop is a prop name or None. Note that for backward-compatibility reasons a single (dir, prop) tuple is also allowed. The parameter retired when set to False, returns only live (un-retired) results. When setting it to True, only retired items are returned. If None, both retired and unretired items are returned. The default is False, i.e. only live items are returned by default. The "limit" and "offset" parameters define a limit on the number of results returned and an offset before returning any results, respectively. These can be used when displaying a number of items in a pagination application or similar. A common use-case is returning the first item of a sorted search by specifying limit=1 (i.e. the maximum or minimum depending on sort order). The filter must match all properties specificed. If the property value to match is a list: 1. String properties must match all elements in the list, and 2. Other properties must match any of the elements in the list. This also means that for strings in exact_match_spec it doesn't make sense to specify multiple values because those cannot all be matched. The propname in filterspec and prop in a sort/group spec may be transitive, i.e., it may contain properties of the form link.link.link.name, e.g. you can search for all issues where a message was added by a certain user in the last week with a filterspec of {'messages.author' : '42', 'messages.creation' : '.-1w;'} """ def list(self): """Return a list of the ids of the active items in this class. """ def count(self): """Get the number of items in this class. If the returned integer is 'numitems', the ids of all the items in this class run from 1 to numitems, and numitems+1 will be the id of the next item to be created in this class. """ # Manipulating properties: def getprops(self): """Return a dictionary mapping property names to property objects. """ def addprop(self, **properties): """Add properties to this class. The keyword arguments in 'properties' must map names to property objects, or a TypeError is raised. None of the keys in 'properties' may collide with the names of existing properties, or a ValueError is raised before any properties have been added. """ def getitem(self, itemid, cache=1): """ Return a Item convenience wrapper for the item. 'itemid' must be the id of an existing item of this class or an IndexError is raised. 'cache' indicates whether the transaction cache should be queried for the item. If the item has been modified and you need to determine what its values prior to modification are, you need to set cache=0. """ class Item: """ A convenience wrapper for the given item. It provides a mapping interface to a single item's properties """ Hyperdatabase Implementations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Hyperdatabase implementations exist to create the interface described in the `hyperdb interface specification`_ over an existing storage mechanism. Examples are relational databases, \*dbm key-value databases, and so on. Several implementations are provided - they belong in the ``roundup.backends`` package. Application Example ~~~~~~~~~~~~~~~~~~~ Here is an example of how the hyperdatabase module would work in practice:: >>> import hyperdb >>> db = hyperdb.Database("foo.db", "ping") >>> db >>> hyperdb.Class(db, "status", name=hyperdb.String()) >>> _.setkey("name") >>> db.status.create(name="unread") 1 >>> db.status.create(name="in-progress") 2 >>> db.status.create(name="testing") 3 >>> db.status.create(name="resolved") 4 >>> db.status.count() 4 >>> db.status.list() [1, 2, 3, 4] >>> db.status.lookup("in-progress") 2 >>> db.status.retire(3) >>> db.status.list() [1, 2, 4] >>> hyperdb.Class(db, "issue", title=hyperdb.String(), status=hyperdb.Link("status")) >>> db.issue.create(title="spam", status=1) 1 >>> db.issue.create(title="eggs", status=2) 2 >>> db.issue.create(title="ham", status=4) 3 >>> db.issue.create(title="arguments", status=2) 4 >>> db.issue.create(title="abuse", status=1) 5 >>> hyperdb.Class(db, "user", username=hyperdb.String(), ... password=hyperdb.String()) >>> db.issue.addprop(fixer=hyperdb.Link("user")) >>> db.issue.getprops() {"title": , "status": , "user": } >>> db.issue.set(5, status=2) >>> db.issue.get(5, "status") 2 >>> db.status.get(2, "name") "in-progress" >>> db.issue.get(5, "title") "abuse" >>> db.issue.find("status", db.status.lookup("in-progress")) [2, 4, 5] >>> db.issue.history(5) [(, "ping", "create", {"title": "abuse", "status": 1}), (, "ping", "set", {"status": 2})] >>> db.status.history(1) [(, "ping", "link", ("issue", 5, "status")), (, "ping", "unlink", ("issue", 5, "status"))] >>> db.status.history(2) [(, "ping", "link", ("issue", 5, "status"))] For the purposes of journalling, when a Multilink property is set to a new list of items, the hyperdatabase compares the old list to the new list. The journal records "unlink" events for all the items that appear in the old list but not the new list, and "link" events for all the items that appear in the new list but not in the old list. Roundup Database ---------------- The Roundup database layer is implemented on top of the hyperdatabase and mediates calls to the database. Some of the classes in the Roundup database are considered issue classes. The Roundup database layer adds detectors and user items, and on issues it provides mail spools, nosy lists, and superseders. Reserved Classes ~~~~~~~~~~~~~~~~ Internal to this layer we reserve three special classes of items that are not issues. Users """"" Users are stored in the hyperdatabase as items of class "user". The "user" class has the definition:: hyperdb.Class(db, "user", username=hyperdb.String(), password=hyperdb.String(), address=hyperdb.String()) db.user.setkey("username") Messages """""""" E-mail messages are represented by hyperdatabase items of class "msg". The actual text content of the messages is stored in separate files. (There's no advantage to be gained by stuffing them into the hyperdatabase, and if messages are stored in ordinary text files, they can be grepped from the command line.) The text of a message is saved in a file named after the message item designator (e.g. "msg23") for the sake of the command interface (see below). Attachments are stored separately and associated with "file" items. The "msg" class has the definition:: hyperdb.Class(db, "msg", author=hyperdb.Link("user"), recipients=hyperdb.Multilink("user"), date=hyperdb.Date(), summary=hyperdb.String(), files=hyperdb.Multilink("file")) The "author" property indicates the author of the message (a "user" item must exist in the hyperdatabase for any messages that are stored in the system). The "summary" property contains a summary of the message for display in a message index. Files """"" Submitted files are represented by hyperdatabase items of class "file". Like e-mail messages, the file content is stored in files outside the database, named after the file item designator (e.g. "file17"). The "file" class has the definition:: hyperdb.Class(db, "file", user=hyperdb.Link("user"), name=hyperdb.String(), type=hyperdb.String()) The "user" property indicates the user who submitted the file, the "name" property holds the original name of the file, and the "type" property holds the MIME type of the file as received. Issue Classes ~~~~~~~~~~~~~ All issues have the following standard properties: =========== ========================== Property Definition =========== ========================== title hyperdb.String() messages hyperdb.Multilink("msg") files hyperdb.Multilink("file") nosy hyperdb.Multilink("user") superseder hyperdb.Multilink("issue") =========== ========================== Also, two Date properties named "creation" and "activity" are fabricated by the Roundup database layer. Two user Link properties, "creator" and "actor" are also fabricated. By "fabricated" we mean that no such properties are actually stored in the hyperdatabase, but when properties on issues are requested, the "creation"/"creator" and "activity"/"actor" properties are made available. The value of the "creation"/"creator" properties relate to issue creation, and the value of the "activity"/ "actor" properties relate to the last editing of any property on the issue (equivalently, these are the dates on the first and last records in the issue's journal). Roundupdb Interface Specification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The interface to a Roundup database delegates most method calls to the hyperdatabase, except for the following changes and additional methods:: class Database: def getuid(self): """Return the id of the "user" item associated with the user that owns this connection to the hyperdatabase.""" class Class: # Overridden methods: def create(self, **propvalues): def set(self, **propvalues): def retire(self, itemid): """These operations trigger detectors and can be vetoed. Attempts to modify the "creation", "creator", "activity" properties or "actor" cause a KeyError. """ class IssueClass(Class): # Overridden methods: def __init__(self, db, classname, **properties): """The newly-created class automatically includes the "messages", "files", "nosy", and "superseder" properties. If the 'properties' dictionary attempts to specify any of these properties or a "creation", "creator", "activity" or "actor" property, a ValueError is raised.""" def get(self, itemid, propname): def getprops(self): """In addition to the actual properties on the item, these methods provide the "creation", "creator", "activity" and "actor" properties.""" # New methods: def addmessage(self, itemid, summary, text): """Add a message to an issue's mail spool. A new "msg" item is constructed using the current date, the user that owns the database connection as the author, and the specified summary text. The "files" and "recipients" fields are left empty. The given text is saved as the body of the message and the item is appended to the "messages" field of the specified issue. """ def nosymessage(self, itemid, msgid): """Send a message to the members of an issue's nosy list. The message is sent only to users on the nosy list who are not already on the "recipients" list for the message. These users are then added to the message's "recipients" list. """ Default Schema ~~~~~~~~~~~~~~ The default schema included with Roundup turns it into a typical software bug tracker. The database is set up like this:: pri = Class(db, "priority", name=hyperdb.String(), order=hyperdb.String()) pri.setkey("name") pri.create(name="critical", order="1") pri.create(name="urgent", order="2") pri.create(name="bug", order="3") pri.create(name="feature", order="4") pri.create(name="wish", order="5") stat = Class(db, "status", name=hyperdb.String(), order=hyperdb.String()) stat.setkey("name") stat.create(name="unread", order="1") stat.create(name="deferred", order="2") stat.create(name="chatting", order="3") stat.create(name="need-eg", order="4") stat.create(name="in-progress", order="5") stat.create(name="testing", order="6") stat.create(name="done-cbb", order="7") stat.create(name="resolved", order="8") Class(db, "keyword", name=hyperdb.String()) Class(db, "issue", fixer=hyperdb.Multilink("user"), keyword=hyperdb.Multilink("keyword"), priority=hyperdb.Link("priority"), status=hyperdb.Link("status")) (The "order" property hasn't been explained yet. It gets used by the Web user interface for sorting.) The above isn't as pretty-looking as the schema specification in the first-stage submission, but it could be made just as easy with the addition of a convenience function like Choice for setting up the "priority" and "status" classes:: def Choice(name, *options): cl = Class(db, name, name=hyperdb.String(), order=hyperdb.String()) for i in range(len(options)): cl.create(name=option[i], order=i) return hyperdb.Link(name) .. index:: schema; detectors design single: detectors; design of single: pair: detectors; auditors single: pair: detectors; reactors Detector Interface ------------------ Detectors are Python functions that are triggered on certain kinds of events. These functions are placed in a special directory which exists just for this purpose. Importing the Roundup database module also imports all the modules in this directory, and the ``init()`` function of each module is called when a database is opened to provide it a chance to register its detectors. There are two kinds of detectors: 1. an `auditor` is triggered just before modifying an item 2. a `reactor` is triggered just after an item has been modified When the Roundup database is about to perform a ``create()``, ``set()``, ``retire()``, or ``restore`` operation, it first calls any *auditors* that have been registered for that operation on that class. Any auditor may raise a *Reject* exception to abort the operation. If none of the auditors raises an exception, the database proceeds to carry out the operation. After it's done, it then calls all of the *reactors* that have been registered for the operation. .. index:: detectors; interface specification single: detectors; register single: auditors; class registration method single: reactors; class registration method pair: class methods; audit pair: class methods; react Detector Interface Specification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``audit()`` and ``react()`` methods register detectors on a given class of items:: class Class: def audit(self, event, detector, priority=100): """Register an auditor on this class. 'event' should be one of "create", "set", "retire", or "restore". 'detector' should be a function accepting four arguments. Detectors are called in priority order, execution order is undefined for detectors with the same priority. """ def react(self, event, detector, priority=100): """Register a reactor on this class. 'event' should be one of "create", "set", "retire", or "restore". 'detector' should be a function accepting four arguments. Detectors are called in priority order, execution order is undefined for detectors with the same priority. """ Auditors are called with the arguments:: audit(db, cl, itemid, newdata) where ``db`` is the database, ``cl`` is an instance of Class or IssueClass within the database, and ``newdata`` is a dictionary mapping property names to values. For a ``create()`` operation, the ``itemid`` argument is None and newdata contains all of the initial property values with which the item is about to be created. For a ``set()`` operation, newdata contains only the names and values of properties that are about to be changed. For a ``retire()`` or ``restore()`` operation, newdata is None. Reactors are called with the arguments:: react(db, cl, itemid, olddata) where ``db`` is the database, ``cl`` is an instance of Class or IssueClass within the database, and ``olddata`` is a dictionary mapping property names to values. For a ``create()`` operation, the ``itemid`` argument is the id of the newly-created item and ``olddata`` is None. For a ``set()`` operation, ``olddata`` contains the names and previous values of properties that were changed. For a ``retire()`` or ``restore()`` operation, ``itemid`` is the id of the retired or restored item and ``olddata`` is None. Detector Example ~~~~~~~~~~~~~~~~ Here is an example of detectors written for a hypothetical project-management application, where users can signal approval of a project by adding themselves to an "approvals" list, and a project proceeds when it has three approvals:: # Permit users only to add themselves to the "approvals" list. def check_approvals(db, cl, id, newdata): if "approvals" in newdata: if cl.get(id, "status") == db.status.lookup("approved"): raise Reject("You can't modify the approvals list " "for a project that has already been approved.") old = cl.get(id, "approvals") new = newdata["approvals"] for uid in old: if uid not in new and uid != db.getuid(): raise Reject("You can't remove other users from " "the approvals list; you can only remove " "yourself.") for uid in new: if uid not in old and uid != db.getuid(): raise Reject("You can't add other users to the " "approvals list; you can only add yourself.") # When three people have approved a project, change its status from # "pending" to "approved". def approve_project(db, cl, id, olddata): if ("approvals" in olddata and len(cl.get(id, "approvals")) == 3): if cl.get(id, "status") == db.status.lookup("pending"): cl.set(id, status=db.status.lookup("approved")) def init(db): db.project.audit("set", check_approval) db.project.react("set", approve_project) Here is another example of a detector that can allow or prevent the creation of new items. In this scenario, patches for a software project are submitted by sending in e-mail with an attached file, and we want to ensure that there are text/plain attachments on the message. The maintainer of the package can then apply the patch by setting its status to "applied":: # Only accept attempts to create new patches that come with patch # files. def check_new_patch(db, cl, id, newdata): if not newdata["files"]: raise Reject("You can't submit a new patch without " "attaching a patch file.") for fileid in newdata["files"]: if db.file.get(fileid, "type") != "text/plain": raise Reject("Submitted patch files must be " "text/plain.") # When the status is changed from "approved" to "applied", apply the # patch. def apply_patch(db, cl, id, olddata): if (cl.get(id, "status") == db.status.lookup("applied") and olddata["status"] == db.status.lookup("approved")): # ...apply the patch... def init(db): db.patch.audit("create", check_new_patch) db.patch.react("set", apply_patch) Command Interface ----------------- The command interface is a very simple and minimal interface, intended only for quick searches and checks from the shell prompt. (Anything more interesting can simply be written in Python using the Roundup database module.) .. index:: roundup-admin; command line specification Command Interface Specification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A single command, ``roundup-admin``, provides basic access to the hyperdatabase from the command line:: roundup-admin help roundup-admin get [-list] designator[, designator,...] propname roundup-admin set designator[, designator,...] propname=value ... roundup-admin find [-list] classname propname=value ... See ``roundup-admin help commands`` for a complete list of commands. Property values are represented as strings in command arguments and in the printed results: - Strings are, well, strings. - Integers/Numbers are displayed the same as strings. - Booleans are displayed as 'Yes' or 'No'. - Date values are printed in the full date format in the local time zone, and accepted in the full format or any of the partial formats explained above. - Link values are printed as item designators. When given as an argument, item designators and key strings are both accepted. - Multilink values are printed as lists of item designators joined by commas. When given as an argument, item designators and key strings are both accepted; an empty string, a single item, or a list of items joined by commas is accepted. When multiple items are specified to the roundup-admin get or roundup-admin set commands, the specified properties are retrieved or set on all the listed items. When multiple results are returned by the roundup-admin get or roundup-admin find commands, they are printed one per line (default) or joined by commas (with the -list) option. .. index:: roundup-admin; usage in scripts Usage Example ~~~~~~~~~~~~~ To find all messages regarding in-progress issues that contain the word "spam", for example, you could execute the following command from the directory where the database dumps its files:: shell% for issue in `roundup-admin find issue status=in-progress`; do > grep -l spam `roundup-admin get $issue messages` > done msg23 msg49 msg50 msg61 shell% Or, using the -list option, this can be written as a single command:: shell% grep -l spam `roundup-admin get \ \`roundup-admin find -list issue status=in-progress\` messages` msg23 msg49 msg50 msg61 shell% E-mail User Interface --------------------- The Roundup system must be assigned an e-mail address at which to receive mail. Messages should be piped to the Roundup mail-handling script by the mail delivery system (e.g. using an alias beginning with "|" for sendmail). Message Processing ~~~~~~~~~~~~~~~~~~ Incoming messages are examined for multiple parts. In a multipart/mixed message or part, each subpart is extracted and examined. In a multipart/alternative message or part, we look for a text/plain subpart and ignore the other parts. The text/plain subparts are assembled to form the textual body of the message, to be stored in the file associated with a "msg" class item. Any parts of other types are each stored in separate files and given "file" class items that are linked to the "msg" item. The "summary" property on message items is taken from the first non-quoting section in the message body. The message body is divided into sections by blank lines. Sections where the second and all subsequent lines begin with a ">" or "|" character are considered "quoting sections". The first line of the first non-quoting section becomes the summary of the message. All of the addresses in the To: and Cc: headers of the incoming message are looked up among the user items, and the corresponding users are placed in the "recipients" property on the new "msg" item. The address in the From: header similarly determines the "author" property of the new "msg" item. The default handling for addresses that don't have corresponding users is to create new users with no passwords and a username equal to the address. (The web interface does not permit logins for users with no passwords.) If we prefer to reject mail from outside sources, we can simply register an auditor on the "user" class that prevents the creation of user items with no passwords. The subject line of the incoming message is examined to determine whether the message is an attempt to create a new issue or to discuss an existing issue. A designator enclosed in square brackets is sought as the first thing on the subject line (after skipping any "Fwd:" or "Re:" prefixes). If an issue designator (class name and id number) is found there, the newly created "msg" item is added to the "messages" property for that issue, and any new "file" items are added to the "files" property for the issue. If just an issue class name is found there, we attempt to create a new issue of that class with its "messages" property initialized to contain the new "msg" item and its "files" property initialized to contain any new "file" items. Both cases may trigger detectors (in the first case we are calling the set() method to add the message to the issue's spool; in the second case we are calling the create() method to create a new item). If an auditor raises an exception, the original message is bounced back to the sender with the explanatory message given in the exception. Nosy Lists ~~~~~~~~~~ A standard detector is provided that watches for additions to the "messages" property. When a new message is added, the detector sends it to all the users on the "nosy" list for the issue that are not already on the "recipients" list of the message. Those users are then appended to the "recipients" property on the message, so multiple copies of a message are never sent to the same user. The journal recorded by the hyperdatabase on the "recipients" property then provides a log of when the message was sent to whom. Setting Properties ~~~~~~~~~~~~~~~~~~ The e-mail interface also provides a simple way to set properties on issues. At the end of the subject line, ``propname=value`` pairs can be specified in square brackets, using the same conventions as for the roundup-admin ``set`` shell command. Web User Interface ------------------ The web interface is provided by a CGI script that can be run under any web server. A simple web server can easily be built on the standard CGIHTTPServer module, and should also be included in the distribution for quick out-of-the-box deployment. The user interface is constructed from a number of template files containing mostly HTML. Among the HTML tags in templates are interspersed some nonstandard tags, which we use as placeholders to be replaced by properties and their values. Views and View Specifiers ~~~~~~~~~~~~~~~~~~~~~~~~~ There are two main kinds of views: *index* views and *issue* views. An index view displays a list of issues of a particular class, optionally sorted and filtered as requested. An issue view presents the properties of a particular issue for editing and displays the message spool for the issue. A view specifier is a string that specifies all the options needed to construct a particular view. It goes after the URL to the Roundup CGI script or the web server to form the complete URL to a view. When the result of selecting a link or submitting a form takes the user to a new view, the Web browser should be redirected to a canonical location containing a complete view specifier so that the view can be bookmarked. Displaying Properties ~~~~~~~~~~~~~~~~~~~~~ Properties appear in the user interface in three contexts: in indices, in editors, and as search filters. For each type of property, there are several display possibilities. For example, in an index view, a string property may just be printed as a plain string, but in an editor view, that property should be displayed in an editable field. The display of a property is handled by functions in the ``cgi.templating`` module. Displayer functions are triggered by ``tal:content`` or ``tal:replace`` tag attributes in templates. The value of the attribute provides an expression for calling the displayer function. For example, the occurrence of:: tal:content="context/status/plain" in a template triggers a call to:: context['status'].plain() where the context would be an item of the "issue" class. The displayer functions can accept extra arguments to further specify details about the widgets that should be generated. Some of the standard displayer functions include: ========= ============================================================== Function Description ========= ============================================================== plain display a String property directly; display a Date property in a specified time zone with an option to omit the time from the date stamp; for a Link or Multilink property, display the key strings of the linked items (or the ids if the linked class has no key property) field display a property like the plain displayer above, but in a text field to be edited menu for a Link property, display a menu of the available choices ========= ============================================================== See the `customisation`_ documentation for the complete list. Index Views ~~~~~~~~~~~ An index view contains two sections: a filter section and an index section. The filter section provides some widgets for selecting which issues appear in the index. The index section is a table of issues. Index View Specifiers """"""""""""""""""""" An index view specifier looks like this (whitespace has been added for clarity):: /issue?status=unread,in-progress,resolved& keyword=security,ui& :group=priority,-status& :sort=-activity& :filters=status,keyword& :columns=title,status,fixer The index view is determined by two parts of the specifier: the layout part and the filter part. The layout part consists of the query parameters that begin with colons, and it determines the way that the properties of selected items are displayed. The filter part consists of all the other query parameters, and it determines the criteria by which items are selected for display. The filter part is interactively manipulated with the form widgets displayed in the filter section. The layout part is interactively manipulated by clicking on the column headings in the table. The filter part selects the union of the sets of issues with values matching any specified Link properties and the intersection of the sets of issues with values matching any specified Multilink properties. The example specifies an index of "issue" items. Only issues with a "status" of either "unread" or "in-progres" or "resolved" are displayed, and only issues with "keyword" values including both "security" and "ui" are displayed. The items are grouped by priority arranged in ascending order and in descending order by status; and within groups, sorted by activity, arranged in descending order. The filter section shows filters for the "status" and "keyword" properties, and the table includes columns for the "title", "status", and "fixer" properties. Associated with each issue class is a default layout specifier. The layout specifier in the above example is the default layout to be provided with the default bug-tracker schema described above in section 4.4. Index Section """"""""""""" The template for an index section describes one row of the index table. Fragments protected by a ``tal:condition="request/show/"`` are included or omitted depending on whether the view specifier requests a column for a particular property. The table cells are filled by the ``tal:content="context/"`` directive, which displays the value of the property. Here's a simple example of an index template:: Sorting """"""" String and Date values are sorted in the natural way. Link properties are sorted according to the value of the "order" property on the linked items if it is present; or otherwise on the key string of the linked items; or finally on the item ids. Multilink properties are sorted according to how many links are present. Issue Views ~~~~~~~~~~~ An issue view contains an editor section and a spool section. At the top of an issue view, links to superseding and superseded issues are always displayed. Issue View Specifiers """"""""""""""""""""" An issue view specifier is simply the issue's designator:: /patch23 Editor Section """""""""""""" The editor section is generated from a template containing ``tal:content="context//"`` directives to insert the appropriate widgets for editing properties. Here's an example of a basic editor template::
As shown in the example, the editor template can also include a ":note" field, which is a text area for entering a note to go along with a change. When a change is submitted, the system automatically generates a message describing the changed properties. The message displays all of the property values on the issue and indicates which ones have changed. An example of such a message might be this:: title: Polly Parrot is dead priority: critical status: unread -> in-progress fixer: (none) keywords: parrot,plumage,perch,nailed,dead If a note is given in the ":note" field, the note is appended to the description. The message is then added to the issue's message spool (thus triggering the standard detector to react by sending out this message to the nosy list). Spool Section """"""""""""" The spool section lists messages in the issue's "messages" property. The index of messages displays the "date", "author", and "summary" properties on the message items, and selecting a message takes you to its content. Access Control -------------- At each point that requires an action to be performed, the security mechanisms are asked if the current user has permission. This permission is defined as a Permission. Individual assignment of Permission to user is unwieldy. The concept of a Role, which encompasses several Permissions and may be assigned to many Users, is quite well developed in many projects. Roundup will take this path, and allow the multiple assignment of Roles to Users, and multiple Permissions to Roles. These definitions are not persistent - they're defined when the application initialises. There will be three levels of Permission. The Class level permissions define logical permissions associated with all items of a particular class (or all classes). The Item level permissions define logical permissions associated with specific items by way of their user-linked properties. The Property level permissions define logical permissions associated with a specific property of an item. Access Control Interface Specification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The security module defines:: class Permission: ''' Defines a Permission with the attributes - name - description - klass (optional) - properties (optional) - check function (optional) The klass may be unset, indicating that this permission is not locked to a particular hyperdb class. There may be multiple Permissions for the same name for different classes. If property names are set, permission is restricted to those properties only. If check function is set, permission is granted only when the function returns value interpreted as boolean true. The function is called with arguments db, userid, itemid. ''' class Role: ''' Defines a Role with the attributes - name - description - permissions ''' class Security: def __init__(self, db): ''' Initialise the permission and role stores, and add in the base roles (for admin user). ''' def getPermission(self, permission, classname=None, properties=None, check=None): ''' Find the Permission exactly matching the name, class, properties list and check function. Raise ValueError if there is no exact match. ''' def hasPermission(self, permission, userid, classname=None, property=None, itemid=None): ''' Look through all the Roles, and hence Permissions, and see if "permission" exists given the constraints of classname, property and itemid. If classname is specified (and only classname) then the search will match if there is *any* Permission for that classname, even if the Permission has additional constraints. If property is specified, the Permission matched must have either no properties listed or the property must appear in the list. If itemid is specified, the Permission matched must have either no check function defined or the check function, when invoked, must return a True value. Note that this functionality is actually implemented by the Permission.test() method. ''' def addPermission(self, **propspec): ''' Create a new Permission with the properties defined in 'propspec'. See the Permission class for the possible keyword args. ''' def addRole(self, **propspec): ''' Create a new Role with the properties defined in 'propspec' ''' def addPermissionToRole(self, rolename, permission): ''' Add the permission to the role's permission list. 'rolename' is the name of the role to add permission to. ''' Modules such as ``cgi/client.py`` and ``mailgw.py`` define their own permissions like so (this example is ``cgi/client.py``):: def initialiseSecurity(security): ''' Create some Permissions and Roles on the security object This function is directly invoked by security.Security.__init__() as a part of the Security object instantiation. ''' p = security.addPermission(name="Web Registration", description="Anonymous users may register through the web") security.addToRole('Anonymous', p) Detectors may also define roles in their init() function:: def init(db): # register an auditor that checks that a user has the "May # Resolve" Permission before allowing them to set an issue # status to "resolved" db.issue.audit('set', checkresolvedok) p = db.security.addPermission(name="May Resolve", klass="issue") security.addToRole('Manager', p) The tracker dbinit module then has in ``open()``:: # open the database - it must be modified to init the Security class # from security.py as db.security db = Database(config, name) # add some extra permissions and associate them with roles ei = db.security.addPermission(name="Edit", klass="issue", description="User is allowed to edit issues") db.security.addPermissionToRole('User', ei) ai = db.security.addPermission(name="View", klass="issue", description="User is allowed to access issues") db.security.addPermissionToRole('User', ai) In the dbinit ``init()``:: # create the two default users user.create(username="admin", password=Password(adminpw), address=config.ADMIN_EMAIL, roles='Admin') user.create(username="anonymous", roles='Anonymous') Then in the code that matters, calls to ``hasPermission`` and ``hasItemPermission`` are made to determine if the user has permission to perform some action:: if db.security.hasPermission('issue', 'Edit', userid): # all ok if db.security.hasItemPermission('issue', itemid, assignedto=userid): # all ok Code in the core will make use of these methods, as should code in auditors in custom templates. The HTML templating may access the access controls through the *user* attribute of the *request* variable. It exposes a ``hasPermission()`` method:: tal:condition="python:request.user.hasPermission('Edit', 'issue')" or, if the *context* is *issue*, then the following is the same:: tal:condition="python:request.user.hasPermission('Edit')" Authentication of Users ~~~~~~~~~~~~~~~~~~~~~~~ Users must be authenticated correctly for the above controls to work. This is not done in the current mail gateway at all. Use of digital signing of messages could alleviate this problem. The exact mechanism of registering the digital signature should be flexible, with perhaps a level of trust. Users who supply their signature through their first message into the tracker should be at a lower level of trust to those who supply their signature to an admin for submission to their user details. Anonymous Users ~~~~~~~~~~~~~~~ The "anonymous" user must always exist, and defines the access permissions for anonymous users. Unknown users accessing Roundup through the web or email interfaces will be logged in as the "anonymous" user. Use Cases ~~~~~~~~~ public - end users can submit bugs, request new features, request support The Users would be given the default "User" Role which gives "View" and "Edit" Permission to the "issue" class. developer - developers can fix bugs, implement new features, provide support A new Role "Developer" is created with the Permission "Fixer" which is checked for in custom auditors that see whether the issue is being resolved with a particular resolution ("fixed", "implemented", "supported") and allows that resolution only if the permission is available. manager - approvers/managers can approve new features and signoff bug fixes A new Role "Manager" is created with the Permission "Signoff" which is checked for in custom auditors that see whether the issue status is being changed similar to the developer example. admin - administrators can add users and set user's roles The existing Role "Admin" has the Permissions "Edit" for all classes (including "user") and "Web Roles" which allow the desired actions. system - automated request handlers running various report/escalation scripts A combination of existing and new Roles, Permissions and auditors could be used here. privacy - issues that are only visible to some users A new property is added to the issue which marks the user or group of users who are allowed to view and edit the issue. An auditor will check for edit access, and the template user object can check for view access. Deployment Scenarios -------------------- The design described above should be general enough to permit the use of Roundup for bug tracking, managing projects, managing patches, or holding discussions. By using items of multiple types, one could deploy a system that maintains requirement specifications, catalogs bugs, and manages submitted patches, where patches could be linked to the bugs and requirements they address. Acknowledgements ---------------- My thanks are due to Christy Heyl for reviewing and contributing suggestions to this paper and motivating me to get it done, and to Jesse Vincent, Mark Miller, Christopher Simons, Jeff Dunmall, Wayne Gramlich, and Dean Tribble for their assistance with the first-round submission. Changes to this document ------------------------ - Added docs for quiet, default_value and required arguments for properties. - Added Boolean, Integer and Number types - Added section Hyperdatabase Implementations - "Item" has been renamed to "Issue" to account for the more specific nature of the Class. - New Templating - Access Controls - Added "actor" property - 'classname' format specified in class Class. .. _customisation: customizing.html code-1a221cf0758b/doc/developers.txt0000644000000000000000000005033114427061170015323 0ustar 00000000000000================== Developing Roundup ================== .. note:: The intended audience of this document is the developers of the core Roundup code. If you just wish to alter some behavior of your Roundup installation, see `Customising Roundup`_. Contents .. contents:: :local: Getting Started --------------- If you are looking for a good first issue, search for `StarterTicket on https://issues.roundup-tracker.org`_. These include issues where Python development, documentation or web design skills are useful. You can continue the conversation using the issue or join the Roundup-devel list to get assistance and verify your planned changes. All development is coordinated through two resources: - roundup-devel mailing list at https://sourceforge.net/projects/roundup/lists/roundup-devel - The issue tracker running at https://issues.roundup-tracker.org/ In addition, the Roundup IRC channel on irc.oftc.net can be accessed via the web interface shown on the Contact page. The channel is logged and the web sites for the logs are shown in the channel topic. You can ask questions and use it to coordinate work discussed using the resources above. Anyone wishing to help in the development of the Roundup Python core may find `Roundup's Design Document`_ and the `implementation notes`_ helpful. People working on Documentation or designing changes to the Web interface don't need to get into the implementation internals. Project Rules ------------- Be polite to others. There is no place for ad hominem attacks. Mostly the project follows Guido's Python Style (though naming tends to be a little relaxed sometimes). In short: - 80 column width code - 4-space indentations Other project rules: - new functionality must be documented, even briefly (so at least we know where there's missing documentation) and changes to tracker configuration must be logged in the upgrading document. - discuss any changes with the other developers on roundup-dev. If nothing else, this makes sure there's no rude shocks. - write unit tests for changes you make (where possible), and ensure that all unit tests run before committing changes. - run flake8_ or pylint_ over changed code. - if you have direct commit access to the repository, subscribe to roundup-checkins to receive checkin notifications from the other developers with write access to the source-code repository. The goal is to have no flake8 issues. Current code has complex functions, some long lines and use of mutable objects in function signatures. Some third party code (e.g. ZPT) vendored into the codebase has more issues. The administrators of the project reserve the right to boot developers who consistently check in code which is either broken or takes the codebase in directions that have not been agreed to. Setting up a Development Environment ------------------------------------ Roundup doesn't require any external libraries. Installing Python 3 and its core libraries is enough to get Roundup running. The easiest way to work with Roundup is to clone the repository. Roundup is developed using the `Mercurial distributed version control system (DVCS)`_ [1]_. It is `hosted at Sourceforge`_. See https://www.roundup-tracker.org/code.html for details. If you are used to git, Mercurial's `command equivalence table`_ can help. Most of the concepts from git (except for staging) should be familiar. To clone the repository use:: hg clone http://hg.code.sf.net/p/roundup/code roundup (Yes, that is an http url.) This will create a read only clone (you can't ``hg push``) of the repo. Changes you make can be attached as patches (created using ``hg diff``) to tickets in our `issue tracker`_. See https://www.roundup-tracker.org/code.html for URL's and directions on getting read write access which allows pushing to the repository. Once you have your clone, you can run ``python3 ./demo -b sqlite`` and get a working Roundup server. This will start the server using the ``sqlite`` backend. The code is in the ``roundup`` subdirectory. Submitting Changes ------------------ Most small changes can be submitted as patches through the `issue tracker`_ or sent to `Roundup-devel mailing list`_. Your account on sourceforge can be set up to allow direct pushes to the repo. Once that is done, using:: hg push https://user@hg.code.sf.net/p/roundup/code or hg push ssh://user@hg.code.sf.net/p/roundup/code will commit your changes. Other Resources - CI, Code Coverage ----------------------------------- Roundup has a `copy of the mercurial repository on GitHub`_. It is updated manually after every few commits to the Mercurial repository. Updates trigger the CI pipeline which happens on two services: 1. `GitHub Actions`_. It runs Docker container scans using Anchore as well as security scans for dependencies using CodeQL. Also it runs the test suite on multiple versions of Python. 2. `TravisCI`_ is also used to run CI. It runs the test suite on multiple Python versions. It also provides alpha and development Python releases faster than GitHub. GitHub actions upload coverage statistics to both `CodeCov`_ and `Coveralls`_. TravisCI only uploads to CodeCov. We run our own issue tracker so we can dogfood the code. As a result, we do not use GitHub issues. Pull requests are grudgingly accepted. They have to be exported and applied to the Mercurial repository. This is time consuming so patches attached to an issue in our tracker are preferred. Debugging Aids -------------- See :doc:`debugging `. Internationalization Notes -------------------------- How stuff works: 1. Strings that may require translation (messages in human language) are marked in the source code. This step is discussed in `Marking Strings for Translation`_ section. 2. These strings are all extracted into Message Template File ``locale/roundup.pot`` (_`POT` file). See `Extracting Translatable Messages`_ below. 3. Language teams use POT file to make Message Files for national languages (_`PO` files). All PO files for Roundup are kept in the ``locale`` directory. Names of these files are target locale names, usually just 2-letter language codes. `Translating Messages`_ section of this chapter gives useful hints for message translators. 4. Translated Message Files are compiled into binary form (_`MO` files) and stored in ``locale`` directory (but not kept in the source code repository, as they may be easily made from PO files). See `Compiling Message Catalogs`_ section. 5. Roundup installer creates runtime locale structure on the file system, putting MO files in their appropriate places. 6. Runtime internationalization (_`I18N`) services use these MO files to translate program messages into language selected by current Roundup user. Roundup command line interface uses locale name set in OS environment variable ``LANGUAGE``, ``LC_ALL``, ``LC_MESSAGES``, or ``LANG`` (in that order). Roundup Web User Interface uses language selected by currently authenticated user. Additional details may be found in `GNU gettext`_ and Python `gettext module`_ documentation. `Roundup source distribution`_ includes POT and PO files for message translators, and also pre-built MO files to facilitate installations from source. Roundup binary distribution includes MO files only. .. _GNU gettext: GNU gettext package ~~~~~~~~~~~~~~~~~~~ This chapter is full of references to GNU `gettext package`_. GNU gettext is a "must have" for nearly all steps of internationalizing any program, and it's manual is definitely a recommended reading for people involved in `I18N`_. There are GNU gettext ports to all major OS platforms. Windows binaries are available from `GNU mirror sites`_. Roundup does not use GNU gettext at runtime, but it's tools are used for `extracting translatable messages`_, `compiling message catalogs`_ and, optionally, for `translating messages`_. Note that ``gettext`` package in some OS distributions means just runtime tools and libraries. In such cases gettext development tools are usually distributed in separate package named ``gettext-devel``. Marking Strings for Translation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Strings that need translation must be marked in the source code. Following subsections explain how this is done in different cases. If translatable string is used as a format string, it is recommended to always use *named* format specifiers:: _('Index of %(classname)s') % locals() This helps translators to better understand the context of the message and, with Python formatting, remove format specifier altogether (which is sometimes useful, especially in singular cases of `Plural Forms`_). When there is more than one format specifier in the translatable format string, named format specifiers **must** be used almost always, because translation may require different order of items. It is better to *not* mark for translation strings that are not locale-dependent, as this makes it more difficult to keep track of translation completeness. For example, string ```` (in ``index()`` method of the request handler in ``roundup_server`` script) has no human readable parts at all, and needs no translations. Such strings are left untranslated in PO files, and are reported as such by PO status checkers (e.g. ``msgfmt --statistics``). Command Line Interfaces ~~~~~~~~~~~~~~~~~~~~~~~ Scripts and routines run from the command line use "static" language defined by environment variables recognized by ``gettext`` module from Python library (``LANGUAGE``, ``LC_ALL``, ``LC_MESSAGES``, and ``LANG``). Primarily, these are ``roundup-admin`` script and ``admin.py`` module, but also help texts and startup error messages in other scripts and their supporting modules. For these interfaces, Python ``gettext`` engine must be initialized to use Roundup message catalogs. This is normally done by including the following line in the module imports:: from i18n import _, ngettext Simple translations are automatically marked by calls to builtin message translation function ``_()``:: print(_("This message is translated")) Translations for messages whose grammatical depends on a number must be done by ``ngettext()`` function:: print(ngettext("Nuked %i file", "Nuked %i files", number_of_files_nuked)) Deferred Translations ~~~~~~~~~~~~~~~~~~~~~ Sometimes translatable strings appear in the source code in untranslated form [#note_admin.py]_ and must be translated elsewhere. Example:: for meal in ("spam", "egg", "bacon"): print(_(meal)) In such cases, strings must be marked for translation without actual call to the translating function. To mark these strings, we use Python feature of automatic concatenation of adjacent strings and different types of string quotes:: strings_to_translate = ( ''"This string will be translated", ""'me too', ''r"\raw string", ''""" multiline string""" ) .. [#note_admin.py] In current Roundup sources, this feature is extensively used in the ``admin`` module using method docstrings as help messages. Web User Interface ~~~~~~~~~~~~~~~~~~ For Web User Interface, translation services are provided by Client object. Action classes have methods ``_()`` and ``gettext()``, delegating translation to the Client instance. In HTML templates, translator object is available as context variable ``i18n``. HTML templates have special markup for translatable strings. The syntax for this markup is discussed at `ZPTInternationalization`_. (Originally documented at http://dev.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/ZPTInternationalizationSupport which is now gone.) Roundup translation service currently ignores values for ``i18n:domain``, ``i18n:source`` and ``i18n:target``. Template markup examples: * simplest case::
Say no more!
this will result in msgid ``"Say no more!"``, with all leading and trailing whitespace stripped, and inner blanks replaced with single space character. * using variable slots::
And now...
No.
THE LARCH
Msgid will be: ``"And now...
No.${slideNo}
THE LARCH"``. Template rendering will use context variable ``number`` (you may use any expression) to put instead of ``${slideNo}`` in translation. * attribute translation::