In this blog post I will share some of the experienced gained in unit testing a large Gtk application written in Python. Some of it only applies to Python, but most of the concepts are possible to implement in other languages.

First a bit of background. As most software projects Stoq started out with no unittests at all. That changed some 5-6 years ago when unit tests for the core business logic were added. Almost all application state of Stoq is stored in PostgreSQL, accessed via the psycopg2 database adapter and Storm, an excellent ORM written by Gustavo Niemeyer.

The current (late march 2013) code coverage of the domain classes is above 90%,
we’ve been trying to increase this gradually, but it’s a lot of work to gap the
remaining part and have never really been the focus.

The business logic, which we refer to as the domain classes, are conceptually
easy to test and has been documented elsewhere so I will not go in great detail in here.

While the domain classes are arguable the most important parts to test in the Stoq case, for instance doing payment related calculations wrong would be disastrous, we’ve always had problems of errors creeping up in level above the UI layer on top of it.

Sometimes around August last year I decided to investigate if anything could be done to
easily increase the coverage of code in the UI layer, which involves Gtk+/PyGTK.

My first approach was something like:

 dialog = PaymentEditor()
 assertEqual(dialog.value_widget.get_text(), "0")
 assertEqual(dialog.description_widget.get_text(), "")

PaymentEditor() constructs the widget tree, either manually or via a glade file,
creates an empty database domain object (Payment) and attaches it to the dialog.

Now, let’s test with real values, create a Payment and attach it to the form:

payment = Payment(value=10.0, description="New Payment")

Create another dialog showing the domain object:

dialog = PaymentEditor(payment)

And verify that the dialog has the right values set:

 assertEqual(dialog.value_widget.get_text(), "10.0")
 assertEqual(dialog.description_widget.get_text(), "New Payment")

Okay, so far so good.
This has increased the code coverage and we test opening an empty dialog,
which is used when creating a new payment and we’re testing.

There are a couple of problems with this approach though:

• While working fine for 1 dialog and 2 fields, it’s not really scalable to huge assistents (~8 pages, 50+ widgets)
• We’re only testing a limited subset, we want to test everything that’s visible. Labels, entries, packing, selection, columns and kiwi extensions such as validation, mandatory state, input masks.

So instead of checking each widget individually, let’s serialize the whole widget tree,
into what we call a UI-test, it looks like this:

GtkDialog(main_dialog.toplevel): title='Edit Details of "New Payment"', hidden
  GtkVBox(main_dialog._main_vbox):
    GtkVBox(main_dialog.vbox, expand=True, fill=True):
      GtkEventBox(main_dialog.main, expand=True, fill=True):
        GtkAlignment(alignment):
          GtkTable():
            ProxyLabel(description_lbl): 'Description:'
            ProxyEntry(description): 'New payment'
            ProxyLabel(value_lbl): 'Value:'
            ProxyEntry(value): '10.00', insensitive

Okay, the UI-test format has a couple of improvements:

• basic widget hierarchy is tested (g_type_name, g_type_parent)
• content of GtkEntry/GtkLabel are included
• widget sensitivity (via insensitive) and visibility (via hidden)
• packing options (expand, fill)

We also do some Python magic to include the variable names of the widgets, so it’s easier to read and understand which widget is is which, this is the value in parenthesis in the name. The dot notation means that the widget reference is stored in a sub instance of the test dialog.

This is similar to HTML page testing, where you save the rendered content of your web application to disk. We plug this into our testing infrastructure so that we have a call like:

self.check_dialog(dialog, "test.uitest")

Which serializes the widget tree to a string and compares it to the previous run, which is stored in the source code repository. If any changes are made to the widget tree, we show a diff against current and last known state, eg:

======================================================================
FAIL: stoqlib.gui.test.test_missingitemsdialog:TestMissingItemsDialog.test_confirm
----------------------------------------------------------------------
Traceback (most recent call last):
[snip]
failed:
test.uitest --- 
test.uitest +++ 
test.uitest @@ -1,9 +1,9 @@
test.uitest -dialog: ConfirmSaleMissingDialog
test.uitest +dialog: MissingItemsDialog
test.uitest GtkDialog(toplevel): title='Missing items', hidden
test.uitest GtkVBox(_main_vbox):
test.uitest GtkVBox(vbox, expand=True, fill=True):
test.uitest GtkEventBox(header):
test.uitest - GtkLabel(): '<b>The following items don&apos;t have enough stock to confirm the sale</b>'
test.uitest + GtkLabel(): '<b>The following items don&apos;t have enough stock to confirm.</b>'
test.uitest GtkEventBox(main, expand=True, fill=True):
test.uitest ObjectList(_klist):
test.uitest column: title='Product', expand

In the test above which is a real error that happened to yesterday, a GtkLabel() changed and someone forgot to update the UI-test, which is a simple rm + rerunning the test.

A real UI-test for our PaymentEditor can be found here.

In the format above, we’re also including the complete domain objects, which may contain attributes that are not shown in the interface, but nevertheless important to test.

The implementation for this, which is pretty generic and couple be used for any Gtk application currently lives inside Stoq and can be found here:

https://github.com/stoq/stoq/blob/master/stoqlib/gui/uitestutils.py

At some point I need to sit down and move it out of Stoq and put in an external library that can easily be used by other Python applications, as it currently a bit tied to kiwi/Stoq.

And the real PaymentEditor test can be found here, which shows how to use the internal API can be found here.

One important aspect of successful UI-testing involves mocking, as we need to be able to fake state to be able to test all code. But that’s for a separate posting as this is already getting a bit long.

So in summary, our total coverage before starting to do UI testing was around 35%, essentially no interfaces were tested. 8 month after first being introduced we have written 419 different UI-tests and the coverage of our currently at 78%. Remember this is a project that has 80k lines+ of Python code.

The main effect of this is that we’ve reduced the amount of QA needed before doing new releases and we have a lot more confidence that things keep working when doing large refactorization.

Of course we need to use the great WebKitGtk library. Unfortunately we cannot use the introspection based bindings as this needs to work on Gtk+ 2.18 and PyGTK 2.17 which were shipped in the last Ubuntu LTS release.

WebView will do all the html/css/js parts. It’s almost as simple as a normal GtkTextView, add it to a scrolled window, load the content and off you go.

The first challange comes when you want to open http:// links in your normal browser, instead of handling them in your webkit. To do that you need to listen the navigation-policy-decision-requested signal ignore certain requests. You don’t actually need to use http protocols, you can invent any url which is parsable.

Next problem is AJAX, to write a proper asynchronous widget you don’t want to reload the whole page when something changes. Since we cannot implement our own protocols in the old libsoup bindings shipped for PyGTK we need to run our own http server. That is good for other reasons as well, we can do heavy IO such as database queries in there without actually blocking the user interaction.

When we need to execute scripting in gtk we just call web_view_execute_script() which will just execute a piece of javascript. For instance,

view.execute_script(“document.title = $(‘fc-header-title’).text()” is a actual line in Stoq, it sets the window title based on calendar header title from the dom.

Going the other direction is a bit uglier, the only way of communcation I found out was opening new urls, so I implemented an application specific domain which opens a dialog or some other action within the gtk application.

I know that some of these tricks are already outdated, in newer webkitgtk versions you should write your own libsoup handlers, use the gobject dom bindings for communication, but I didn’t have these options when writing this post

TL;DR

• Listen to the ::navigation-policy-decision-requested to implement your own uri handling if you can’t do it via libsoup.
• Run a separate daemon process which will serve as an internal webserver, so AJAX calls work and won’t block on IO.
• Use web_view_execute_script() for Gtk->Javascript communication
• Use window.location = “customprotocol://”  for Javascript ->Gtk communication
• Make the javascript parts work in a normal browser so the normal developer tools can be used
• If possible, use a newer version of WebKitGtk and avoid all of this.

Xan and the other webkit hackers will probably look at me in disgust for telling you how to do all of these dirty hacks

Calendar application

It’s now possible to list payments, purchase orders and client calls in a graphical view:

It might look familiar, it uses the fantastic javascript library fullcalendar. We really wanted to use a normal GtkWidget for the calendar but it would have been a lot more work to rip out half of evolution. If there are any other options that can match fullcalendars functionallity there we’d be open to switching as embedding WebKit, jQuery and fullcalendar in a Gtk+ application is not ideal.

Configurable keyboard shortcuts

This is something that has been requested many times over the years. It makes it easier to remap the keyboard bindings use often to other keys, such as the function keys. There’s still an open task to redo all the existing keybindings that aren’t uniform enough.

Configurable form fields

Some companies does not use all the form fields (fax anyone?) that we show per default and Stoq know has a configuration interface where you can make fields non-mandatory and even hide them if you don’t wish to see them. Perfect for the first steps of localization.

New manual

One of our interns rewrote old docbook manual to mallard, and it looks beatiful and is now well integrated in the application. You can find the online version here. It involved removing a lot of screenshots and text. It’ll be easier to update the manual in the future if there aren’t any screenshots. He also fixed the interface, there are now various help buttons in the application that goes to a help section describing that part.

Localization support

It’s now possible to configure some of the fields that are specific to each region/country. The only thing that made it into this release was company identification number (Brazil: CNPJ, Sweden: Organisationnr, US: Employer Identification Number). But person identification number and list of states has landed in the code repository since the release. We still need someone to step up and start doing the actual localization for this, be the hero of the day and download Stoq and start localizing it!

Boleto Bancário (Bank invoice)

Brazilian banks supports a kind of invoice with a barcodes/numbers, called boleto bancário. It’s semi-standardadized, most of the data is similar, but you need to special case each bank that should be supported. There are two kinds, with and without cobrança (for eventually sending to a collection agency). There are a couple of 100 active banks and about 15 major ones. Stoq currently supports 7: Banco do Brasil, Banco Real, Banco Santander, Banco Bradesco, Caixa Econômica, Banrisul, Banco Itaú. All without cobrança though, support for that will come in a future release.

Call for volunteers

Stoq has initially been targeting the Brazilian market, since that what’s close to the current development team. But there is now longer an excuse for not trying to use it. We can barely handle the legal part of Brazil and we’d need volunteer help to make it possible to use in other countries. We’re very proud of the application so we wouldn’t want to stop you just because you live outside of Brazil!

So, why don’t you grab the code and get started, it’s all python (and a tiny bit of javascript) and shouldn’t be hard to get started.

Don’t be discouraged by the web site and manual is only in Portuguese, we use gettext and rosetta and the code is modular and easy to understand.

We’ll need a lot of work to support localization in different countries such as: company/person formats, states, taxes and other things we don’t know about yet, let us know and we’ll try to find a solution.

Just send me a mail or come in on our new shiny web chat: http://chat.stoq.com.br/ (aka #stoq on freenode)

It’s written in Gtk/Python/PostgreSQL and is available for Linux/Windows. Free software (GPL), stable, fast and beautiful!

New for this release is the reorganization of the applications to always fit into one window (conceptually similar to evolution/outlook) and a new application selector which should make it easier to switch between different parts of the app.

It’s fully translated and can be used outside of Brazil, even though a few small details are oriented around the fiscal requirements set by the brazilian government.

There’s a livecd available on Ubuntu, so you can just download it and try it out.

For more information see http://www.stoq.com.br/

I had a great time and we talked about my involvement in GNOME, Stoq together with a bunch of other random topics, check it out at:

http://www.castalio.info/johan-dahlin-stoq-gestor-comercial-livre/

But first some brief history. Back in the old PyGTK and pygtk-codegen days all functions were created at compile time generating all the Python<->C wrappers that you would ever need. This is problematic for various reasons:

First of all, the litl got uncovered and released a couple of weeks ago. The whole litl software platform is heavily dependent on the introspection parts and on the gjs, the spidermonkey javascript bindings. It’s been a great experience of easily being able to move code between javascript and C as we see fit. Adding a new method to a GObject class in C is just a make a away from making it available to the javascript bindings. Lucas and Scott have written more about the technical parts of the litl.

Gjs has recently gained callback support, which has been kind of a personal pet peeve for me. Making sure that functions and methods with callbacks are possible to call from a language binding has always been difficult. GClosure sort of solves that problem, unfortunately very few libraries actually use them. Most of the tricky work of landing callback support in Gjs was done by Maxim Ermilov, rock on!

The Python bindings are also moving along pretty quickly. Tomeu Vizoso and Simon Van Der Linden has been working aggressively on making them solid. The rumor tells me that Tomeu has the whole OLPC Sugar interface running on top the new bindings, with significant speed and memory improvments!

Elliot Smith at Intel blogged about his experiences on using Gjs on Moblin developing a clutter application. Zach Goldberg writes about his experiences on adding callback support to the python bindings.

More and more libraries are adding introspection support, the latest ones are libgda and moblin toolkit.

If you’re interested in language bindings or other uses of the introspection data, come by and help us out.

Oh, I almost forgot to say hi to Planet Python. Andrew was kind enough to add me there. My name is Johan Dahlin and I’m mostly working on Python related to GNOME, especially, Python bindings for GObject and Gtk., and Kiwi + Stoq of course. The introspection project is intended to make it easier to write language bindings for the Gnome stack, not only Python ones.

Code Generator move

One of the most important parts of the gtk python bindings lies within the code generator. This piece of software which was mostly written by James Henstridge, Gustavo Carneiro and I takes an API definition file (in scheme s-expression style) and output C glue which tell Python how to call the API in a library. For various reasons the code generator is mostly intended to be used by a GObject based library. It was originally written for PyGTK but has since been used in a number of other placs, most noticable gst-python contains a fork which various modifications. It has recently moved from the PyGTK package to PyGObject and can now easily be used by GObject based library which wishes to use it without depending on GTK+.

GIO

The principal motivation for moving the code generator from PyGTK to PyGObject was to be able to create GIO bindings. An initial set of bindings has been created for GIO. They are not complete yet, quite a bit of the API which takes async result parameters has not yet been written, but it should be perfectly usable by applications already. Please use it and let me know if there is any methods missing and I will gladly wrap them for you

GLib

One of the complaints over there years has been that it’s kind of weird to type gobject.MainLoop(), gobject.io_add_watch() etc, since these functions are not really related to GObject. Complain no more, they have now moved to a new module called just glib. It’s been quite hard to come up with any reasonable use cases for third party packages using glib but not gobject. I guess one of them is python bindings for Qt wishing to integrate with the python bindings for Gtk+. Qt already provides (optional) glib/mainloop integration, so perhaps this would be useful to them.

Porting to Python 3

This weekend I ported the glib and gobject modules over to be compilable against Python 3.0b2. It has been done in such a way that the source code is compatible with both the 2.x and 3.0 APIs of Python. To be able to do that I was forced to add a number of quite intrusive macros which takes care of the hard work. I’ve only managed to reach the point where it is possible to import the gobject module. The code generator and the gio bindings has not yet been ported. Even less has the testsuite been run, so this is just moderatly useful at this point. As part of porting this I had to make the python support in automake be compatible with python 3. Patches against git head of automake can be found here.