Experiments in public

The twist on the experiment

2021-02-23T12:36:00.000-08:00

Preface

Having resurrected this blog for the purpose of helping find myself a job (again), some things are going to be different.

The principle thing is a switch in focus, less on the technical and more on the people side, corresponding with how my career has evolved over the past decade. I probably will still dig into some technical topics, both out of interest and because it's easier to find opportunities to solve technical problems in the wilds of the internet, but it's important I also demonstrate, as best as I am able, that I have some understanding of organizational dynamics and human issues.

The secondary thing that's different is nearly a consequence of the first thing. In short, I will try to use this space to promote the efforts of those less well-represented in tech than myself. I will still surely include some of my own opinions, but my experience is that my opinion is much less likely to provide unique value than that of, say, a black woman.

Compared to the last 5-7 years, where in general I've sought to avoid injecting myself into discussions, I am going to use this space to highlight and discuss relevant events. Again, where I can I will populate those discussions by including quotes from and links to others thoughts on the subjects, but when I believe I can provide value by collating and commenting on these subjects, I will.

I will be writing about:

People - Process - Product

When I was working on my resume recently, I shrunk it down about 75%, and I wanted a pithy way to convey where my priorities were at this point. Both to put it out into the world, as an expression of what I was looking for, but also because it represented a clear shift in how I approached "work" over the past decade. While process has always been a significant part of the work I do in the tech world, it's generally been in service of the delivery of a particular product. Yes, I spent plenty of time optimizing processes for people, but that wasn't where the focus was. So a big part of resurrecting this blog is also about representing that shift to a focus on people.

People

Writing about people here is likely to be the most difficult, in some ways. I don't think there's a bug tracker for people problems, where I could just volunteer to try to help them, then write about it. But one of the ways that I think I can incorporate writing about people into my efforts here is through the relatively introspective manner that I tend to approach problems. That is, through documented analysis of my own process and behaviour in approaching problems, I can demonstrate at least some personal understanding of the dynamics involved.

Besides that, I will write about patterns and individual instances of workplace people problems that I encounter, primarily on Twitter. I will do my best to gather evidence demonstrating the harms that are occurring, and highlighting opportunities to mitigate those harms. I expect to lean heavily on the work of others here, and if I allow the focus of this type of writing to be on me, please call me out on it. That said, I will incorporate examples and stories from my own experience, as appropriate.

Process

Writing about process is something I generally enjoy, and while I'm used to doing that in a professional setting where I have long-term plans for the processes in question, I don't think I'll have a problem finding many, many relevant opportunities when writing about people or products.

Product

While definitely the lesser focus overall in how I'm approaching work, I will demonstrate delivery of high quality products throughout here. Both in the written work, and any accompanying projects, completeness and correctness will be held to a high standard.

Enhancing Google Keep

To that end, my first project here is going to be building an opinionated task management tool on top of Google Keep. It will allow me to write about many of the topics relevant to the kind of work I'm seeking, while also scratching a very old itch for me. :)

The Automator's Manifesto, Part Two

2021-02-22T12:18:00.003-08:00

This was written quite a while ago, but I never got around to publishing it. So here you go!

Proactive Transparency

One of the most important things any automator can do is provide confidence to stakeholders in the project. Confidence can only develop if stakeholders understand the project, and see meaningful improvement. As such, it behooves the automator to make it easy to understand the state of the project. This is particularly relevant when it comes to actual test results, as even successful results that have no audience are essentially useless.

The term I came up with for this aspect of an automators role is "proactive transparency", based on the idea that it's not enough to simply run tests and call it a day. It's a good start, but publishing the results of those tests in a way so as to maximize their audience is where real value starts to develop.

Entropy Factors

The most challenging automation task of my career came at Research in Motion, where I put together a device lab to test over the air OS software updates (RIM was actually the first to have this available, as I understand). This was a hugely complex endeavour, covering a range of devices with different capabilities, moving between OS versions that could vary greatly. But even more significant to our testing were the externalities we were dependent on.

For instance, knowing what builds were available for updating required manual scraping of a development version of an application maintained by a different team, with no formal connection. This application, in turn, was loosely connected to the servers which devices would contact to identify what updates might apply to them. We frequently encountered issues where a test was queued, but when the device navigated to the update screen, the desired build wasn't found.

These, and many others, came to be known as our "entropy factors". I'm probably misusing entropy, but here the phrase means things outside of your control that could impact the results of your tests. Understanding what these are for a given automation project is one of the most important planning exercises you can do.

Measuring Everything

At my first GTAC, Patrick Copeland said "measure everything" (he probably said "at Google we measure everything", but that wasn't what I wrote down.) This was an inspiring credo for a then new-to-the-field automator, and I spent many years trying to live up to it. But it's wrong, or at least deceptive.

Assuming you have the resources to make it possible, measuring everything is still going to require implementation time. But what is much more important is that it is going to increase the scale of measuring output, while doing nothing to identify which metrics mean success. It might feel great to have every system metric, API call, user session and what-have-you carefully collected, but it does nothing on it's own to make your project better. That's an oversimplification, as having this all can increase stakeholder confidence initially, but after a couple of scares, that will fade.

You need to commit to measurements that correspond to your desired outcomes for the automation project. That doesn't mean you shouldn't collect a ton of other metrics if your tooling can do it for free, but you shouldn't surface them by default. Keep clear what you are measuring and why, and add cases to that as you grow.

Renewing the Experiment

2021-02-22T04:59:00.002-08:00

I started this blog about a decade ago to help me find work.

Once I found work, it began to atrophy rather quickly.

Now, I need to find work again. :)

So here's a placeholder, saying that this blog is coming back to life. I've got a handful of unpublished drafts from the past several years that I may polish and publish, and I'll write about my job search otherwise.

My fling with iOS, or "I like my gardens unwalled"

2017-03-12T18:01:00.001-07:00

Somewhat of a digression from the usual content of this blog (either programming or nothing), and ground that has been well-trod in many other places. But here I am, writing it anyway.

TL;DR

I switched to an iPhone 7 2 months ago. It was really cool, but I don't like it and am switching back.

Origin

I've never owned an iPhone. My wife has had a 4, 5, 6, and 7, and other family members have had many others.

I worked at BlackBerry back when it was called Research in Motion, starting in 2008, around the launch of the Bold 9000. Before then, I didn't even own a cellphone (I had an awesome Sanyo in 2001, but it broke and I never replaced it).

While at RIM I changed phones every 6 months or so. Luckily, I got to more or less avoid the Storm and follow-ups (though I was responsible for implementing touch support in our test automation framework), so I got very used to the BlackBerry keyboards. I still think the keyboard in the original Bold is a work of art.

From when RIM and I parted ways in 2011 until the end of last year, I used almost entirely Androids, except for a while where I enjoyed the LG Quantum on Windows. I actually really liked that phone, and Windows there worked really well. Like the commercials emphasized, you could very quickly get things done and put your phone back down. Part of this was because there was no reason to keep looking at your phone, because it couldn't do much more than essential communication functions, but at the time I found that simplicity quite charming.

But Android...

Well, my first experience with Android was a week with the Galaxy Nexus. It was pretty cool at the time, but I returned it due to the expense.

But when the Nexus 4 came-a-callin, I was smitten. It was a beautiful piece of hardware that ran all kinds of cool things, did very little to get in my way, and cost half of what my wife's phone did. Sure, the camera was mostly crap and the battery wasn't great, but I loved that thing.

The Nexus 4 was followed by the Moto X. I picked mine up on Kijiji, and loved the active display and passable camera.

I followed the Moto X with the Droid Turbo, which was nearly a perfect phone. Swiping my hand over to check notifications is still one of my favourite phone interactions, and the battery would go two days before needing a charge (which only took 90 minutes anyway). The camera, despite the resolution, was bad most of the time, and as a Verizon exclusive that I exported to Canada, I had to put up with useless apps and slooooow updates. Still, it lasted me nearly two years before the screen got damaged (which was supposed to be under warranty, but Verizon and Moto figured they could get away with screwing a Canadian).

Which brings me to my last (and near future) Android, the LG G4. Battery life was a big let down from the Droid, but the camera was very good, and it had some other neat features. I cracked the screen on it in 3 months. I ordered a replacement screen, but while I was waiting for it to arrive, an opportunity to go iPhone arose.

The iPhone

For all that my phones have rarely been top-of-the-line, I read multiple reviews of every flagship, and the pace at which Apple has pushed forward their CPU was amazing. With the 6S I was tempted to jump ship just for that reason, and seeing how far they pushed it further in the 7 made it impossible to resist.

The iPhone 7 is the most attractive piece of hardware to touch, ever. The weight is perfect, it's sleek, and it responds to you in a very physical way.

It is blazing fast at practically everything, and the screen is second-to-none.

The camera is super-fast, and generally very good, though I would argue it trades blows with the G4 in some areas.

Force touch is cool, and makes some things nicer to do.

I like being able to ask Siri to time things for me. Very cool.

I get through a day with 40% left on the battery, easily. But that's not the whole story...

My wife can't get through the day without having to recharge, with the same phone. Because she uses it for Facebook. I generally don't use mine for anything like that. In fact, I use my iPhone the way I used my Windows Phone. When I need it, it's there, and then I put it away.

But it's the only piece of Apple in my ecosystem, and that means a lot of the things I want to do are just a little bit inconvenient, so I don't do them. My life is on Google's cloud, and my work is on Microsoft's. These things are available in a pinch from my iPhone, but my Surface is substantially better for either of them, and not that much harder to tote around.

Specific examples of annoying things:
- I must use a case, because it's $$$, which detracts substantially from the aesthetic appeal. I've never used a case before
- Google maps navigation cuts out over bluetooth
- Notifications are among the worst on any phone I've ever had. Better since I disabled the volume switches affecting them, but some apps just fail to notify me consistently (might be on the apps, but they worked fine on Android)
- I've got relatively large hands, and I cannot seem to get comfortable with the keyboard. Some of this might be the case, some might be that I've developed a preference for Swype, who knows.
- Siri is really dumb. I don't know if it's somehow related to my lack of Apple ecosystem, but other than timing things or prank questions, I've given up talking to her
- No headphone jack is just stupid. Yes I have two dongles in the house, but mostly I just carry two sets of headphones now. Especially stupid if they're going to move to USB-C.
- Chromecast (the only input to my TVs) control is inconsistent. Some apps I can change volume with buttons, others within the app, others I need to open Google Home to control volume. Probably an app issue, but not a problem I had before.
- My fitbit stopped pairing with it, and one of my vehicles will only pair for a day or so before needing to removed, bluetooth toggled off and on, and re-added.
- The UI on some of Apples apps is surprisingly bad. Sometimes when I change settings, I click "back" in the top left, other times I hit "save" in the top right. Some apps require you to do both these things to effect a change.

Summary

Subjectively, the feeling is that I need to do things Apple's way, and I'm not interested in that approach. So all that lovely power never gets called upon. A lot of my issues are little things, many of which there may be fixes for, but my experience is that unlike Android, there's not a lot you can do to fix these little things.

So it's back to the G4 for me, with its fresh screen. I won't use a case, and I'll probably break it again. But it's not getting Nougat anyway, so maybe it will be time to trade up again by then. :)

Software Gardening in practice

2015-10-17T03:29:00.000-07:00

Software gardening, as expressed by Hunt/Thomas, Atwood, etc uses the metaphor to deal with how a software project is alive, and is constantly being changed by factors outside of the control of the developers. You can plant a seed, and water it, but you can’t control the sunlight, the bugs, bees, etc. As it grows, you can weed the garden, removing features that are not used but taking up resources, add fertilizer, change the plants to better reflect your needs (why didn’t someone tell me Joe was allergic to cucumbers!?).

I want to expand on this idea, as I believe there are several valuable insights that can be derived from adopting this view. In this post I will explore this concept from three different perspectives: the project itself, a software developer, and a client that wants to get some software developed.

The Unseen Environment

As a particular software environment is like a garden, there are many factors that the garden grows out of. There are obvious factors, like the composition of the soil, and the likely availability of water, and sunlight. But there are also more subtle, “living” factors, that are going to be part of the environment of your garden, yet generally operate independently.

One of the most fascinating things I’ve learned recently is the concept of mycorrhizal networks. These are fungal layers that exist within the soil that serve to break down waste matter and distribute molecules that plants depend on for nutrition. It is generally unseen, but is a huge factor in the ability of plant ecosystems to thrive. Generally much better known, but still somewhat unseen, are the pollinators. Birds, bees, and butterflies allow plants to reproduce, and thus stimulate genetic diversity.

So, above and below, there is an environment that allows plant life to thrive. A similar role is played in software development by the ecosystem that exists around both the software, and the developers.
All developers and software libraries come to a project with a history that is shaped through the absorption of “byproducts” from previous projects. These can take the form of learned code snippets, software structure conventions, and many other things. I would liken this to the way the mycorrhizal network digests the output from various past biological processes and produces raw nutrients to begin new ones.

In a similar way, open/shared source cultures could be compared to pollinators. Moving from project to project with very few barriers, a developer can distribute efficient techniques rapidly, effecting the projects and other developers. This has resulted in the establishment of numerous best practices, to the point where high-profile open source projects are the gold standards for software quality. Another interesting way in which open source diversity is like genetic diversity is in the trend of multiple competing software projects springing up with different attributes tailored to their environments. These can continue for very long periods, sharing “genes” with each other but still continuing as distinct “species”.

In some large organizations, the software ecosystem is distinct enough to effect the software that is produced there, but at this point, almost everyone is dependent on Open Source. It produces specific codebases that underly most every modern project, such as Linux, Apache, OpenSSL, GCC, Boost, Python, etc. But it also establishes the best practices and standards for software development that are used by individual developers, whatever they are working on. Conversely, bug reports come from commercial projects that make Open Source stronger, and developer experiences there continue to inform and refine the state of the art. There are also cases where it is beneficial for commercial development to open source certain pieces of their software, to expose them to that wider diversity, and encourage contributions that will help meet business goals (though like natural evolution, predicting the future of an open source project is far from a science).

Tying that back together, that means that for a developer, every project you work on contributes to this larger community, even when the code itself is closed. Because it changes you, and it changes the conversations you have with others, and how you work to solve problems in the future. As a former user of the Zope Web Application Server, I have watched other users and contributors define the cutting edge of web software (Tarek Ziade, Chris McDonough, Alex Gaynor, and Martijn Faasen, among others).

Garden Development as a Service

As someone who works on software, your value is not the creation and delivery of a product, but rather the cultivation of an environment. Even when you are employed delivering products, there’s no such thing as a final release, just something that is able to meet the needs of the moment. Needs will change, the ecosystem will change, you will change, and new opportunities will arise. How might this change how you approach a single project, and your career?

In terms of approaching a particular project, it means you need to understand the landscape in order to effectively execute on the vision. You need to know what kind of software is already there, and what the state of third party libraries are in that particular domain. You need to plan for the inevitable bugs and maintenance. The actual implementation flows out of these concerns, not the other way around.

From a career standpoint, I should first confess that, well, I’m not a gardener. However the image I have of them is that they don’t get annoyed that all the cool gardens are already cultivated. As long as you have a willingness to work, gardens are going to need cultivation. Likewise, as long as you have a willingness to develop it, there will always be available work developing software. Under this philosophy, nothing ever reaches the point that it cannot be improved or does not require maintenance.

Gardening Agency

When we have a need for some software, and we attempt to consider the effort that will be required, we usually think about the features that we want the software to have. The gardening metaphor suggests this is a very limited way to view this. If we think of the features of a particular garden as “has Roses and Violets, and produces Tomatoes”, we can quickly understand that a good deal more information is required to get a reasonable estimate of the effort that will be necessary.

Is the garden in Hawaii, Alaska, or the Central Park Zoo? What if there’s already a garden there? What's the condition of the soil, or local animal life?

Translating these questions over to software, we can see that the environment a piece of software needs to "live" in can impact the effort required to develop it as much or more than the features it needs to have. For instance, if you are looking to add accounting features for exporting financial data to your partners, the effort could vary tremendously depending on the existence and maturity of the open source software that exists to work with data in the formats they use.

Just as working with a professional landscaper will produce dramatically different results than simply purchasing plants, establishing a relationship with someone who can look at your situation holistically is key to a successful software project. They know the questions that need to be asked, the process for finding the answers, and they know how to build out a development team that will take that reality into account.

It's yer data! - how Google secured its future, and everyone else's

2014-08-29T16:31:00.000-07:00

Dear Google,

This is a love letter and a call to action.

I believe we stand at a place where there is a unique opportunity in managing personal data.

There is a limited range of data types in the universe, and practically speaking, the vast majority of software works with a particularly tiny fraction of them.

People, for example. We know things about them.

Names, pictures of, people known, statements made, etc.

Tons of web applications conceive of these objects. Maybe not all, but probably most have some crossover. For many of the most trafficked apps, this personal data represents a very central currency. But unfortunately, up until now we've more or less been content with each app having it's own currency, that is not recognized elsewhere.

You can change that. You can establish a central, independent bank of data, owned by users and lent to applications in exchange for functionality. The format of the data itself will be defined and evolved by an independent agency of some sort.

There are two core things this will accomplish.

1) It will open up a whole new world of application development free from ties to you, Facebook, Twitter, etc.

2) It will give people back ownership of their data. They will be able to establish and evolve an online identity that carries forward as they change what applications they use.

Both of these have a dramatic impact on Google, as they allow you to do what you do best, building applications that work with large datasets, while at the same time freeing from you concerns that you are monopolizing people's data.

A new application world

When developing a new application, you start with an idea, and then you spend a lot of time defining a data model and the logic required to implement that idea on that data model. If you have any success with your application, you will need to invest further in your data model, fleshing it out, and implementing search, caching, and other optimizations.

In this new world, all you would do is include a library and point it at an existing data model. For the small fraction of data that was unique to your application, you could extend the existing model. For example:

from new_world import Model, Field

BaseUser = Model("https://new_world.org/users/1.0")

class OurUser(BaseUser):
    our_field = Field("our_field", type=String)

That's it. No persistence (though you could set args somewhere to define how to synchronize), no search, no caching. Now you can get to actually building what makes your application great.

Conceivably, you can do it all in Javascript, other than identifying the application uniquely to the data store.

And you can be guaranteed data interoperability with Facebook, Google, etc. So if you make a photo editing app, you can edit photos uploaded with any of those, and they can display the photos that are edited.

Securing our future

People have good reason to be suspicious of Google, Facebook, or any other organization that is able to derive value through the "ownership" of their data. Regardless of the intent of the organization today, history has shown that profit is a very powerful motivator for bad behaviour, and these caches of personal data represent a store of potential profit that we all expect will at some point prove too tempting to avoid abusing.

Providing explicit ownership and license of said data via a third-party won't take away the temptation to abuse the data, but will make it more difficult in a number of ways:

Clear ownership will make unfair use claims much more cut-and-dried
A common data format will make it much easier to abandon rogue applications
Reduced application development overhead will increase the competitive pressure, lowering the chance of a single application monopolizing a market and needing to grow through exploitation of its users data

A gooder, more-productive, Google

By putting people's data back in their hands, and merely borrowing it from them for specific applications, the opportunities for evil are dramatically reduced.

But what I think is even more compelling for Google here is that it will make you more productive. Internally, I believe you already operate similar to how I've described here, but you constantly bump up against limitations imposed by trying not to be evil. Without having to worry about the perceptions of how you are using people's data, what could you accomplish?

Conclusion

Google wants to do no evil. Facebook is perhaps less explicit, but from what I know of its culture, I believe it aspires to be competent enough that there's no need to exploit users data. The future will bring new leadership and changes in culture to both companies, but if they act soon, they can secure their moral aspirations and provide a great gift to the world.

(Interesting aside, Amazon's recently announced Cognito appears to be in some ways a relative of this idea, at least as a developer looking to build things. Check it out.)

PyCon 2014

2014-04-24T07:43:00.000-07:00

I've now been back from PyCon for a week, and I've got some thoughts to share.

Scope

It was huge.

I usually try to memorize everyone's names, and I have some habits that help me with that. But there were so many people, I think that may have fallen apart. :)

A lot of hero worship, as I met, or at least observed from a distance, many people who helped shape my views on software (+Tres Seaver in particular).

Conversely, I managed to avoid running into those attending from my employers (I'm looking at you, +Kenneth Reitz, Sandy Walsh, and probably someone from RIM/BlackBerry).

Diversity

All the promotion of the diversity was terrific. At the same time that it's great to be part of a movement that is markedly more female-friendly then the tech community at large, Jessica McKellar made it clear that we have so much farther to go. As the father of two girls, it's very important to me that we change the culture around technology to emphasize that there's no particular skillset or aptitude that's required for entry.

Software is our world, and we can empower EVERYONE to play a part in shaping it.

Content Overview

I enjoyed the talks that I went to, but I did skip more than I was intending to. I had trouble letting go of work, and there was a lot of content that was relatively beginner focused, or represented tutorials that I knew had high-quality online counterparts, should I need them. I feel like this was a deficiency of my own, and one I hope I handle better if I come back next year.

Meta programming

I've been flirting with creating my own language for a while now, and if I were to do so, it would probably be on top of Python. Thanks to talks by +Allison Kaptur and +Paul Tagliamonte, I feel much more prepared to do so.

Allison provided a brilliant guide to implementing the import mechanism from scratch. Having read +Brett Cannon's blog when he created importlib, I knew there was a huge amount of work that went into getting it right, so it was an intimidating area. Yet in 20 minutes Allison walked us through getting something functional.

Paul's talk on Hy was not quite so accessible, but perhaps even more inspiring. The relative ease with which Hy and Python can co-exist within the same project is just awesome, though mucking around with ASTs remains a bit of a scary idea.

Sprints

While I was skipping talks, I consoled myself in the thought that I would really engage during the Sprints (I had a day and half scheduled for these). But I didn't, and while I think that had more to do with me (once again, I worked), I'll share what I think could have been done better, in case anyone else felt the same way.

Technically Sprints started Sunday evening, but I get the feeling that no one was actually interested in running them Sunday evening (or maybe my timing was off). There were a handful of people there, but no clear organization or plan about what was to be worked on.

Monday morning, it was certainly better attended, but it still wasn't inviting. There was a central chart of what rooms contained what projects, but within the rooms there was no indication of who was working on what. From my limited time being involved in or running other short coding sessions, I was also surprised not to see much use of flipcharts or whiteboards.

I guess how I would do it, if I ran it next year (I'm not volunteering, yet), is provide signs for each project to put at their table, and encourage each of them to write out a list of goals for the sprint in a place where it can be publicly examined and crossed off. Perhaps also provide special shirts/hats/badges to the "leader" of each sprint. The experience I would like is for someone to be able to wander in, examine what each project is doing without actually bothering anybody, and then if they find something they think could fit them, to know who to ask.

Misc.

Ansible is what we're using at SFX, and while I've had some experience with it, I have a much more robust appreciation for it, thanks to +Michael DeHaan
Peep should be included in the standard library. Seriously.
Asyncio makes things most people do easier. Bravo!
iPython Notebook is cool, and +Catherine Devlins talk about executable documentation has me itching to try it out.

Conclusion

As someone who has been around the block but doesn't find much time to actually code anymore, I may not be the code audience for PyCon. But I'm still delighted to have finally made it to one, and I'm really tempted to make it a family trip next year.

Rethinking my excuses about hiring for Test Automation

2013-10-11T19:19:00.001-07:00

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."

--Brian Kernighan

For a long time, I've made the excuse that it's hard to hire QA Engineers, etc. because of stigma based on the desired career path going from testing to development, not the other way around. That may be part of it, but I'm beginning to realize that there might be something much more significant.

Being a good Automator is HARD. As Kernighan says, debugging is hard, and automation is all debugging.

Practically speaking, the act of debugging is the investigation of a behaviour to determine it's cause. You know the software does a thing, but you don't know why, and you need to figure out how to stop it (usually). So you run it in a controlled environment and try to get that behaviour to happen.

What Automators do, most of the time, is write software that interfaces with other software. Automation is it's own integration point, but generally speaking, it's a low-priority client, so you frequently have to exploit one or more other integration points in order to get the software to behave as desired for your automation. Usually, what you want the software to do is on the margins of what it was expected to do, so you do a lot of debugging to identify and simulate the conditions that might generate the desired behaviour.

Ok, so that's a lot of horn tooting, but I've met nearly a dozen good Automators, and without exception they are fantastic software developers. That's not to say there aren't other factors that drive people away from automation, but consider this a call to action for those who want to prove their mettle as software craftspeople.

Be an Automator, spend your days debugging everyone else's software, drive the art forward, and enjoy (in the long run) incredible job security!

As an aside, I think the headline quote is also what binds me to the Python community. The Zen of Python guides us to avoid "clever" solutions whenever possible, out of the shared recognizance that in the long run, software spends a lot more time being debugged than developed.

The Automator's Manifesto, Part One

2013-08-21T06:50:00.001-07:00

In my career in the software field, I've had to teach myself about what quality software was. What the differences were between successful software projects, and unsuccessful ones. It's been a long and often frustrating journey, but I feel like I've come out of it with some principles that could be helpful in guiding people in the field.

My principle strength in software development is pattern recognition, especially behavioural. I've used this over the years to refine my approach to software development, by analyzing my own behaviour and that of those around me. For this topic, I've tried to apply that at a higher level to my experience with test automation. What is it about automation that most consistently delivers value? What consistently doesn't? Are there practices can guide us towards the former and away from the latter?

Certainly, to a degree, everything here is derived from common software best practices, but I believe I am able to tailor them in a way that is uniquely suited to the automator. That's why this part is sub-titled…

Automation is Software

(I use the term "automator" as a generalization of test automation developer, quality engineer, software engineer in test, and a dozen other titles).

Apply Best Practices, as best you can

I'm sure we've all heard the statistics about how 50%, 75% or 90% of all software projects fail in some way. If we've been around the block, we may even understand some of the steps to take to improve our chances, commonly called "best practices". Automation projects are almost always directly dependent on some other software project, and when they go down, the automation often goes with them.

So from the start, the odds aren't good with your automation. But then consider that of all software projects, automation seems to be the place where best practices are least… practiced. I'm talking about things like:

Requirements gathering
Structured development process
Documentation
Unit, etc. tests
Source control (I've seen this missing several times)

Not to say that every piece of automation needs all of these, just that an automation project is like any other software project, and if we fail to apply common knowledge about software projects to it, we are shooting ourselves in the foot.
What's more, by practicing these things, you become better equipped to ensure they get applied to the project you are developing automation for, primarily because it can help you establish credibility with the developers working on it. Walking the walk, and all that.

Have a Client

A key aspect of most successful software projects is the presence of an engaged client. As in most aspects, Automation is no different. Someone who understands the value that you are supposed to be delivering is in the best position to help you understand what needs to be done to deliver that value. Do not simply write tests because there was a decree "we need tests for X".

Instead, you need to understand where the desire for tests comes from? Is it that developers are hesitant to make changes because they have no safety net? Is it that there are lots of regressions in production? Is it that there's a performance optimization project underway, and metrics are needed? Each case would suggest a different focus, and there are many more cases.

But in order to get to this point where you have identified the need, and make progress against meeting it, you need a partner who will vet requirements and validate progress. You need a client. This can be a team of developers, a particular developer, someone in management, or others. But they need to understand the value you are expected to deliver, and have the time to confirm that you are doing so.

Fight on the best terrain available to you

As I believe is common to anyone who has been in this field for a while, actually automating tests is not always the best way you can improve the quality of the software. As described in "Test is Dead", many companies have user bases that accept being used for beta testing. Sure, it might not always be so, but if that is the situation you find yourself in, attempting to keep bugs from those beta testers will not be an ideal use of your time.

A robust functional test suite for a rapidly changing product might cost more in maintenance then it delivers in the first few years. A development team without a mature development process will gain little by forcing them to write unit tests. Sometimes there's more value to be gained in doing SysAdmin or DevOps type work, such as creating continuous integration infrastructure. The important thing about being an automator is establishing processes that will make producing higher quality software easier. Whether those are actual automated tests run against software, or merely conventions established to improve communication during development is not important. What is important is that you make producing quality software the easiest thing for the developers.

In order of priority, I would suggest:

Establish a development process
Establish a development environment
Automate tests
Automate monitoring

Each of these makes the following item easier and more valuable.

Summary

Automation is Software, so:

Follow best practices, as best you can
Have a client, who can validate the value you deliver
Understand the terrain of the project you are automating against, and tailor your focus to that

Join me next time as I dive into the value of Proactive Transparency, the danger of Entropy Factors, and the deceptive bliss of Measuring Everything.

Beyond GTAC

2013-04-29T08:45:00.000-07:00

In 2013, in the twilight of my "testing" career, I was able to attend Google Test Automation Conference for the third time. Once again, I was impressed by (and jealous of) many of the solutions presented. But for me at least, I'm not sure I'm the target audience anymore.

About me

Before I get into that, let me expand on what I said above about twilight. The arc of my career is the story of trying to figure out how to use technology to improve people's lives. This began as computer and then network support, making sure people were able to use the tech they already had. From there, I began to write software, for myself and then those who would pay me to do it.

Throughout this process, I have had to constantly analyze and refine my own processes. I made mistakes, researched best practices, and tried again. Everything I do comes from continual experimentation and optimization. My career transitions also come out of this drive for self-improvement.

As I wrote and used more software, and especially assisted others in using it, it became (gradually) clear that the only useful measure of quality was whether it made it easier for users to accomplish their goals.

Most recently, that career has led me to look beyond myself in terms of delivering quality software. I'm now responsible for optimizing the output of a team of fantastic developers. In the majority of situations we currently encounter, writing tests is not the most efficient path towards this.

But enough about me

So, to a certain degree, GTAC may not be for me because I need to take a wider view then many of the participants, even though we share the same goals. But I think there's more to it then that. The people at GTAC are on average likely to be the best test engineers on the planet. Yet with few exceptions, the work they presented would have fit right in with GTAC from 4 or 6 years ago. As a community, we are failing to establish and exploit institutional memory, so we keep having to re-solve the same problems.

As a specific example, Google themselves have an incredible build and release system, as the result of a huge ongoing investment. From what we saw of every non-Google presentation, other companies capabilities for automated testing are hampered by the lack of this kind of infrastructure. Not to imply that Google has any obligation to provide it, but it seems to me that everyone could benefit substantially from more direct collaboration on it. Not to say the tooling in this area doesn't hasn't improved, but it still seems like a problem that every company continues to try to solve in their own way.

Don't get the impression from this that GTAC itself is anything less than fantastic. It has helped me grow so much, and likely raised my expectations for what should be possible. If you have any sort of passion for automated testing, there would be few things more valuable than connecting with this community. Also, despite my gripings, there are some things happening today that simply wouldn't have been possible in the recent past.

About those exceptions

Selenium. Not to be trite about it, but the Selenium project is the biggest change to the test automation community over the past 6 years. It's gone from being a nifty tool for somewhat flaky web automation to a standard (literally) platform upon which we can converse about automation and build other tools. Some examples of this include Appium, which is leveraging it to provide cross-platform mobile app testing, and Marionette, which is integrating Selenium into builds of Firefox OS. It's not the end of the story by a long shot, but having Selenium allows us to elevate our dialogue around test automation.

Misc. Buck, by Facebook, seems like a nifty solution to a problem I almost have (though I'm not sure when +Simon Stewart finds time to stop being funny and start writing software). Almost everyone agrees you should just kill flaky tests, which is good, but I'm not sure it's the endpoint in that conversation. Getting developers to write tests was also a near-unanimous sentiment (and why wouldn't it be in this group?), though unlike Thomas Petersen, I didn't get the impression that it was intended to exclude testers from the development process. Pex4Fun is just cool. The Android UI testing team embedding their automation commands in the build thread seems a bit crazy, but still a really clever way to avoid the whole click-and-sleep cycle that plagues most UI automation.

Organization. Despite having a more limited experience of the Google office, I feel like this was by far the best run conference I have ever been to. Things were on schedule, very smooth, and +Anthony Voellm is one heck of an MC.

Summary

GTAC remains in my mind the place to be if you are interested in doing test automation. I believe there's more that the test automation/software quality community needs to do to grow together, but the fact that this conference isn't all about that is mostly a reflection of my changing priorities, rather than any failings on their part.

Full disclosure, I spent 3 years in automation at ~~RIM~~ BlackBerry, and this was a mobile testing conference, so I'm probably just jaded.

Recursively formatting a dictionary of strings

2012-03-09T06:19:00.000-08:00

I have a use-case where I'm using a dictionary as a basis for generating a configuration file, and many of the values in the file share pieces, such as URL bases or credentials. I will also be generating configuration files for several different instances from the same dictionary.

I wanted a simple way to define replaceable parts of the dictionary. I made a couple of Google searches, didn't find anything on the first page, so I wrote this:

gist for dict_format.py

I hope it's useful to someone else, or that I find it next time I have this use-case. :)

Shoutout to PyDev

2012-03-08T07:23:00.001-08:00

I've been using PyDev for many years, for Python and Jython development of all kinds. The more recent releases have made some big usability improvements. Specifically, autocompletion works in many more cases, and the built-in refactoring support is proving handy. Rough edges that have bugged me for years are being steadily smoothed away.
So thanks for all the free stuff Fabio! If I was at PyCon I would buy you a drink.

New Year's Python Meme

2011-12-29T07:33:00.000-08:00

As suggested by Tarek, and done by so many others.

1. What’s the coolest Python application, framework or library you have discovered in 2011 ?

Requests. I've dealt with Python's HTTP handling on numerous occasions, and most times found myself constructing an abstraction layer to deal with it. Requests provides this abstraction layer for pretty much all of my use cases. I'll give an honourable mention to Pyramid, because it allowed me to bring together my current use of Pylons and my past experience with Zope.

2. What new programming technique did you learn in 2011 ?

Sorting algorithms ;). On a more serious note, I started actually getting benefit out of Scrum, which is not quite a programming technique but definitely help my programming. On a related note, I learned the benefits of continuous deployment, and some of the techniques one can use to support that.

3. What’s the name of the open source project you contributed the most in 2011 ? What did you do ?

Python Web Client. But I built that myself. I did file an accepted (and quickly fixed) bug in PyDev.

4. What was the Python blog or website you read the most in 2011 ?

Planet Python. Pretty much my go to for Python, though I'm tempted to start reading Python Reddit (I've never used Reddit before).

5. What are the three top things you want to learn in 2012 ?

Chef (and related tools), gevent, Go. I'm also interested in (and more likely to learn) CoffeeScript and Node JS.

6. What are the top software, app or lib you wish someone would write in 2012 ?

A web service mocking library. I've seen some promising signs in this direction, and I have cause to contribute myself to the effort.
A self-replicating Python platform, that can deploy itself from device-to-device relatively transparently. I want to be able to say "run code X on instance Y", and not worry about any of the details of preparing Y or moving X to it.

If you want to participate:

copy-paste the questions and answer to them in your blog
tweet it with the #2012pythonmeme hashtag

An end to the experiments (for now)

2011-10-27T06:10:00.000-07:00

Well, it's been 3 months, and I've gotten a decent number of things written. It took much longer than I was expecting, but I learned a lot of things, and made some good contacts. But most excitingly, I got a job! Last week I accepted a position working for Arc90, most prominently of Readability fame, but they've got plenty of other things going on.

I first heard about them when Kenneth Reitz started working there, and blogged about it. As it turns out, they were looking for someone with test automation experience just about the time I started getting serious about sending my resume around.

I'll be working with the Insight project team, building test automation and reporting tools to ensure confidence in the quality of every release. It's a very different scale from my work at RIM, and I think a very good opportunity for me to succeed or fail entirely on my own merits (I'm going to succeed, btw).

Which brings us to this blog. For the next few months, I don't expect to have much time to commit to writing here. Kenneth has certainly done an admirable job of keeping up blogging (and working on Requests!), so perhaps I will be able to as well. I've done a bit of research on some of the other topics I wanted to write about, and there's a post I think I owe someone for the help they have given me, but Arc90 is going to be my focus.

So this is an end, to this blog as it was. But one of the reasons I chose Arc90 as an employer is their support for blogging and open source work, so I do expect to be back writing before too long. Thanks to everyone who has read and commented on any of this!

Python Web Client 0.2 Release

2011-10-12T09:04:00.000-07:00

I'm pleased to announce the 0.2 release of the Python Web Client. It now meets a bare minimum of functionality to be useful. Specifically, it provides a web UI to allow you to put together and submit HTTP requests, specifying request type, URL, parameters and headers.

You can view a demo here.

Oh, and it's hosted on Github here.

What's new in 0.2?

- Added display of response body
- Added option to select request method
- Added field to add parameters
- Added field to add headers

What does the future hold?

Well, there are a lot of potential ideas for where this could go, both in the original SoCal piggies proposal, and on the Rally site where I've been tracking this (log in with alecmunro+public@gmail.com:Experiments). However, I expect to resume my place among the employed soon, and thus far I haven't had any feedback on this. So barring extreme boredom on my part, this is probably the end of the line until someone forks it.

Python Web Client 0.1

2011-10-04T12:18:00.000-07:00

The release
I've polished a few rough edges, and I'm now (somewhat) proud to release Python Web Client 0.1. It's still lacking in features, but it's got docs and an easy way to start it, and what's more, its pieces work on non-Windows platforms. I've also handicapped it a bit to allow me to put up a hosted version for people to try out, here.

What's next
Well, with this "release", the foundation of the project should be more or less complete (though I don't have it in continuous integration yet). That should make it pretty safe to add features. I'll see what I can put into it over the next few days and get a 0.2 release out, that will hopefully be more useful. If I people are interested, we might make it to 0.3, at which point I'll try to get it on PyPi (it would be nice to have a project up there).

Sidenote
I'm not sure there's a good solution, but I find all the context-switching that goes along with setting up a new project to be a real drain on productivity.

Python Web Client 0.1.dev1

2011-09-26T20:24:00.000-07:00

tl;dr
I built a preliminary version of the Socal Piggies Python Web Client. Take a look at the code and methodology, suggest features/fixes, and otherwise enjoy! There's no live demo because of a gaping security hole.

Intro
First off, I would just like to express my sincere appreciation for everyone who responded to my request for ideas. There were a number of interesting options, but for now, I've decided to build an implementation of the Python Web Client described on the Socal Piggies site. I made this choice because it's a comfortable area for me to work in, it's a tool I can see using, and probably most important, I think I can build a simple version fairly quickly. :)

Design
The first thing I usually do with a new project is retreat to a quiet corner with a notebook and a writing implement. It would be nice to find an electronic way to do this, but so far nothing has come close to what I need in terms of offering a combination or structured and free-form input, along with instant availability.

In this case, I was trying to strip the concept down to the bare essentials. In this case that means:

One page, with two widgets:

Create request (Enter a URL)
Display response (Status Code + Headers)

Startup script that launches a browser to the service

That last one may seem non-essential, but speaks to my philosophy that "delivery is as important as development". In practice, that means that how a client is introduced to functionality is just as important as how well that functionality works. Make it really easy to start using.

Because this is somewhat of a showcase project, there's a couple of other things I pinned on my design list:

Testing (unit, functional, system, jsunit)
Docs (UI + API, published in Sphinx)

While I was putting this together, I also wrote down a whole lot of nice-to-have features for later. You can check them out on the Rally site I am (kind of) using to manage this project. You will need to log in:
alecmunro+public@gmail.com:Experiments

Testing
Because I'm a TDD advocate, let's do that. So first I need to decide what tests I want to have. Since I don't know my code structure at all yet, I'm going to start with system level tests, which I usually define as something that tests the system at the UI level, running very close to how it will run in production:

Visit site, enter URL, press submit, verify results.

Gist-It for test_ui.py

That pretty much does it, and can be done with something like Selenium, unless I also wanted to test the startup script. Doing so would involve using something like Sikuli(which I do love), to observe the state of the desktop, but that might balloon the scope of this project a bit too much. So we are on to functional testing, in this case defined as testing the API of the web service, in as isolated an environment as we can create. So what are we looking at there?

Submit URL, verify response

Probably some variants of this, to test error handling or redirect responses (we are handling those, right?).
What happens if the URL to retrieve is the URL of the webservice itself? Could we experience some nastiness there?

So both of the test types we have addressed so far require an actual connection to another server. We could use something that's always going to be available, like www.google.com, but we aren't really guaranteed a network connection. So for this, I'll write a small web server that can be set to return whatever you want it to.
Gist-It for echo_server.py

This was actually a bit trickier than I anticipated, due to the need to run the server in a separate thread/process, and this bug. Anyway, here's the API tests:

Gist-It for test_apis.py

Ok, so unit tests now. From our earlier tests, it's become pretty clear that the API will have one view, which accepts the details to construct a request (just a URL to start), submits that request, and returns the status code and headers from the response.

Well, that was all really boring. Maybe the jsunit tests will be more interesting? In practice, I probably won't write these before the code, because it still takes me a while to get into the rhythm of writing tests for javascript. I need a bit of trial-and-error.

Create Request Widget:

Enter text and press submit. A call should be made to create the request, and the deferred for that call should be passed to the page.

Gist-It for test_create_request.js

Display Response Widget:

Supply it with various responses, and confirm that they display properly. Probably the most interesting bit of testing of the whole lot.

Gist-It for test_display_response.js

Implementation

Ok, so we have our tests, perhaps. Now, on to the implementation. This part is actually really simple.

There's the Python view:
Gist-It for views.py

and the two Javascript widgets:
Gist-It for create_request.js

Gist-It for display_response.js

Feel free to take a look at the GitHub repository for more details (or check it out to run it).

Documentation
I've left off the documentation for now, both because I wanted to get this up soon, and it's been a while since I started anything with Sphinx, and also because it really doesn't do much yet. I'm also still missing the launch script.

Conclusion
There's lots more work to be done here to make something useful, so I'm taking suggestions. But hopefully this gives you an idea of how you can build a simple and somewhat tested web-app using Pyramid and JQuery. You will notice that it is very testing heavy, probably significantly more than real-world deadlines would allow for. But once you get these tests in place (and a system to run them), they are fairly easy to build on, and can provide a safe container in which to experiment.
For the moment, I'm planning two distinct iterations:

Add docs and launch script, as well as displaying the response body. That will be 0.1
Flesh out the request creation ability, to allow settings headers and parameters. Along with hopefully some fixes/refinements, that will be 0.2

Beyond that, development will depend on whether this is interesting to anyone, so let me know.

Sorting out

2011-09-17T19:16:00.000-07:00

As someone who almost exclusively uses high-level languages, the idea of writing my own sorting algorithm seems a bit silly. However, I can certainly appreciate the algorithms used for sorting, and it's a neat opportunity to try to document and implement these algorithms using methods I am familiar with.

Preface

These implementations are likely to be extremely naive. If you went CS school, you probably learned better implementations in your first year. There is absolutely no reason to use these in production code anywhere. The built-in python sorting is 10-50 times faster than anything I've produced (I'm actually surprised it's not more). If you've got tips about how to speed these up just by changing the code around, feel free to comment, or you can try forking my repository and sending a pull? request. I can't promise to spend much time on it, because this is pretty well-trod territory, but it would be a bit entertaining to see how close we can get.

Quick Sort

My understanding of quicksort is still developing, but based on what I've read on Wikipedia, it can work something like this:

Grab an item in the list, called the "pivot".
Create two lists, representing everything greater than the pivot, and everything less then the pivot.
Repeat steps 1 and 2 on the two sub-lists, returning the a list comprised of the smaller list items, then the pivot, then the larger list items.

In this way, the entire list should get sorted. It's my understanding there are various optimizations that can be done, such as swapping elements instead of creating new lists. It seems to be one of the "best" all around algorithms, though for certain cases, it's performance can be beaten by others.

Gist-it for quick_sort.py

This was the second fastest list implementation I created, at about 1/12th the performance of the built-in sorting.

Merge Sort
Merge sort relies on splitting the list recursively until you have an ordered hierarchy of l item item lists. Then you compare each list with it's neighbour, and return a combined list of the two merged together. Now you have 2-item sorted lists, which you work through, comparing the left-most element to its neighbour's left-most element, and moving the lowest one to a new list, eventually creating a 4-item list.

Gist-it for merge_sort.py

Heap Sort

Heap sort relies on understanding what a "heap" is, in this context. So let's do that.

A heap is a data structure that is tree based, and follows the rule that all children of a node will have a value that is lower than or equal to the value of that node (for a max-heap, for a min-heap the rule is reversed). In practice, heaps are usually implemented within arrays, with the first or last item of the array being the root of the tree.

Now, why do we do have a heap? What's the advantage? Apparently sorting things. :)

In a heap sort, we first build the heap from the list we've been given. Once we have our heap, we know the item with the highest value is at the root of the heap, which is position 0 in the list representing the heap. So now we swap the first and last elements in that list, and either move the last element from the list, or simply treat that part of the heap as already sorted (which means our heap needs to know to ignore it somehow). Then we take the first item, and trickle it down through the heap until the heap is proper again. This consists of comparing it's value with it's children, and swapping it with the higher one of these, until one of these is not higher.

Gist-it for heap_sort.py

Python apparently does have a built-in heap library, which might be worth benchmarking, as heap was my slowest sort implementation.

Bubble Sort

Bubble sort is fairly trivial.

Iterate through the items, except the last one, if an item is greater than the next one, swap their positions. When you reach the end of the list, do it again, unless there were zero swaps.

Gist-it for bubble_sort.py

Insertion Sort

Insertion sort is another trivial sort. I'm really only including it because I think my (unwitting) choice of an insertion sort in an interview may have cost me a job.

Gist-it for insertion_sort.py

Interestingly enough, for my stupid example of sorting a small list of characters, insertion sort is the fastest implementation I created.

Acknowledgements and Conclusion

Not really much to conclude, except that I now have a better understanding how to write a sorting algorithm, and some of the pros and cons of different algorithms. I certainly can't look at a data set and give you the big O case for a given algorithm.

But one thing I do want to draw special attention to is Wikipedia's pages on this subject. They vary a bit in information density, but they were incredibly informative, and make use of numerous methods to help communicate how each sort works. If you really want to understand this stuff, I would definitely start with these pages:

Also, my thought to do this post was sparked by another post on Planet Python that mentioned Insertion Sort, where I recognized the algorithm. I'm not entirely sure, but this may be that post.

Let me build you a web app/service!

2011-09-15T08:25:00.000-07:00

So, according to my introduction, one of my goals for this project is to build a web app or service to demonstrate my skillset in that area. Unfortunately, it's turns out my idea for a web app has already been done, and is quite well-known (on the plus side, I now understand what 4Chan is). At the moment, I'm drawing blanks about what I should build, so I thought perhaps I could crowdsource that job.

So if you have an idea for a web app or service, but not the time or skills to get it started, or if you would simply like to collaborate with someone on it, post a comment describing your idea.

In general, I'm game for pretty much anything, as I'm confident I can learn whatever I need to. Some general parameters though:

It's going to be in Python, built on Pyramid, using SQLAlchemy for models
I will try to strip the idea to it's core, and leave others to make it full-featured
I will try to make an appealing and usable UI, but again, it will be pretty barebones
I'm going to blog about it, host it in github and make the code viewable by everyone
If you want to collaborate:

I'm open to hosting it elsewhere, as long as it's publicly accessible
I will be a stickler in design discussion and code reviews

Finally, I will happily put it under any license you want and grant you and copyright or whatnot, as long as it doesn't prevent me from achieving my objectives vis-a-vis skill demonstration.

Design patterns

2011-09-13T11:17:00.000-07:00

Design patterns are a very broad subject, but as far as I can tell, they are primarily common conventions for structuring software. I've never been taught them, and as far as I can tell, I've used nearly all of them, repeatedly. So for an experienced software developer, I don't believe studying design patterns is likely to improve your coding skills much, if at all. However, it could certainly improve your collaboration skills.

Words Matter
If there's one downside to being self-taught in the field of computer programming (there are several), it would be the difficulty of communicating your ideas with those who have a "standard" education. However good my code may be doesn't matter if I spend all my time trying to explain how I'm going to implement something. Formal computer science comes with a whole boatload of terminology, design patterns among them. Being able to say "use the template method pattern to defer processing to subclasses" is much more concise than "in the superclass method, call methods that are overridden in the subclass, but have only stub implementations in the superclass". Even that verbose description is concise compared to what usually comes up with discussing actual project code.

Template Method
The template method pattern is something that eventually comes in handy to anyone who does a fair amount of object oriented design. It's a response to the problem of how to mix specialized behaviour of subclasses with general behaviour of superclasses. The basic idea is that the superclass implementation of a method will contain mostly calls to other methods of the object, which will be mostly abstract in the superclass, and implemented in the subclasses, which won't implement the method being called.
I recently used this on an email reporting system, which used either HTML or text email depending on the job being reported on. There was a fair amount of commonality between both types, in that they dealt with collecting data from the environment and stuffing it into some kind of template (the use of the word template here is unrelated), then formatting that template into an email and sending it along.
The actual email sending was almost the same for both HTML and text, so with just a condition or two we could handle that entirely in the superclass. However, the data that was sent out varied a bit between the two types, as did the emails themselves.

Gist: "Template Method example"

Note that the implementation above is not ideal, because the subclasses do need a certain awareness of the superclass implementation in order to behave properly.

Mediator

The mediator pattern, in my understanding, mostly deals with managing the flow of communication between objects. It puts a layer between your object and it's intended communication target. I use this pattern frequently for building UIs, most of which I build in JQuery, so that's where I use it.

The way I build UIs in JQuery has been a constantly evolving pattern, so this may change, but right now it gives me the flexibility I need. What I usually do is create an object representing the page itself, as well as objects to represent each of the distinct interface areas, which I am going to call widgets (thought that might be too specific of a term in this context). Each of the widgets takes zero or more callback functions, depending on what they do, and the page object is responsible for seeing that these point to the same things. In some cases, they are methods of other widgets (though in typical javascript fashion, they are anonymous functions that call those methods), but more often they are methods of the page object, which accept the communication and see it dispatched to any other widget(s) that are interested. The page object is almost never more than wiring between the widgets.

The following is probably not syntactically correct. It frequently takes a bit of trial and error before I get into the swing of JavaScript OOP.

Gist: "Mediator Pattern example"

Strategy

The strategy pattern deals with abstracting away algorithms. In situations where a different context might require a different algorithm to achieve the desired results, you apply the strategy pattern. This might sounds complex, but in practice, it is as simple as using different validation functions on different data types, as long as you have a common interface to those functions.

Gist: "Strategy Pattern example"

Adapter

The adapter pattern is one that, thanks to Zope 3, I've been familiar with for a long time. An adapter is an object that "wraps" another object to provide a desired interface. Designing a system based on the adapter pattern requires using capital-I Interfaces for establishing object communication. Then, as necessary, adapters are created to allow the available objects to satisfy those interfaces.
To a certain degree, the adapter pattern takes the concept of polymorphism, traditionally understood as being able to use a superclass' interface to interact with instances of subclasses, and expands it. So you can use the specification of an Interface to interact with any objects that are adaptable to that Interface. To use the traditional Bicycle example, Bicycle might extend Vehicle, and have traditional vehicular methods. However, maybe we want to check what color the bike is, or paint it. Vehicle may not have methods for this, but IPaintable might. Then, with a BicyclePainter adapter, we can adapter our Bicycle instance to this interface. (For the record, I've never found the Bicycle example very useful, and I'm not sure if I'm doing you any favours by using it here).

Gist: "Adapter Pattern example"

Conclusion

So there's some design patterns, and their usage. It's by no means comprehensive, and I'm sure many of you will have encountered these in other contexts where my definitions aren't entirely accurate. That's just fine, because the value of these is not in having rigid development patterns, it's having useful ways to elevate the discussion of your development. I've almost never encountered clearly recognizable design patterns in code I inherited, because patterns are theory, and when applied, the real world makes things start to get messy.

References

Wikipedia: I relied on this for design patterns and their basic descriptions.
"Adapters" chapter from Grok tutorial

Mocking snakes

2011-09-08T11:34:00.000-07:00

Many years ago, I was tasked with improving the performance of a suite of unit tests. They were taking ever longer to run, and were beyond 20 minutes when I started working with them. Needless to say, this meant people rarely ran them.

From Miško Hevery:

What I want is for my IDE to run my tests every time I save the code. To do this my tests need to be fast, because my patience after hitting Cntl-S is about two seconds. Anything longer than that and I will get annoyed. If you start running your tests after every save from test zero you will automatically make sure that your test will never become slow, since as soon as your tests start to run slow you will be forced to refactor your tests to make them faster.

The problem was that every test was descended from a common base class, and that class brought up fake versions of most of the application. Well, mostly fake versions. There was still a lot of I/O and network activity involved in bringing up these fakes.

The solution turned out to be mock objects, JMock in this particular case. For those unfamiliar, mock objects are objects that can stand in for your dependencies, and can be programmed to respond in the particular manner necessary for whatever it is you are testing. So if your network client is supposed to return "oops" every time the network connection times out, you can use a mock to stand in for the network connection, rather than relying on lady fortune to drop the network for you (or doing something terrible, like having code in your test that disables your network interface).

There are a couple of general drawbacks to using mock objects, but the primary one is that a mock object only knows what you tell it. If the interface of your dependencies change, your mock object will not know this, and your tests will continue to pass. This is why it is key to have higher level tests, run less frequently, that exercise the actual interfaces between objects, not just the interfaces you have trained your mocks to have.

The other drawbacks have more to do with verbosity and code structure than anything else. In order for a mock to be useful, you need a way to tell your code under test what dependency it is standing in for. In my code, this tends to lead to far more verbose constructors, that detail every dependency of the object. But there are other mechanisms, which I will explore here.

For a more verbose comparison of mock libraries in a variety of use cases, check this out:

http://garybernhardt.github.com/python-mock-comparison/

Hopefully this post will be a more opinionated supplement to that.

There are a couple of categories of things to mock:

Unreliable dependencies (network, file system)
Inconsistent dependencies (time-dependent functionality)
Performance-impacting dependencies (pickling, hashing functions, perhaps)
Calls to the object under test

The last item is certainly not a necessity to mock, but it does come in handy when testing an object with a bunch of methods that call each other. I'll refer to it as "partial mocking" here.

For this article, I'm going to focus on 4 mock object libraries, Mocker, Flexmock, and Fudge, chosen primarily because they are the ones I have experience with. I also added in Mock, but I don't have much experience with it yet. I believe, from my more limited experience with other libraries, that these provide a decent representation of different approaches to mocking challenges.

I'm going to go through common use cases, how each library handles them, and my comments on that. One important note is that I generally don't (and won't here) differentiate between mocks, stubs, spies, etc.

Getting a mock

Gist: "Getting mock objects from different libraries"

Dependencies are usually injected in the constructor, in a form like the following:
Gist "Verbose dependency specification for mocking"

This is verbose, especially as we build real objects, which tend to have many dependencies, once you start to consider standard library modules as dependencies. :)

NOTE: Not all standard library modules need to be mocked out. Things like os.path.join or date formatting operations are entirely self contained, and shouldn't introduce significant performance penalties. As such, I tend not to mock them out. That does introduce the unfortunate situation where I will have a call to a mocked out os.path on one line, and call to the real os.path on the next:
Gist: "Confusion when not everything is mocked"

This can certainly be a bit confusing at times, but I don't yet have a better solution.

However, it is quite explicit, and avoids the need for a dependency injection framework. Not that there's anything wrong with using such a framework, but doing so steepens the learning curve for your code.

Verifying Expectations

One key aspect of using mock objects is ensuring that they are called in the ways you expect. Understanding how to use this functionality can make test driven development very straightforward, because by understanding how your object will need to work with it's dependencies, you can be sure that the interface you are implementing on those dependencies reflects the reality of how it will be used. For this and more, read Mock Roles Not Objects>, by Steve Freeman and Nat Pryce.

...anyway, verification takes different forms across libraries.
Gist: "Verifying mock expectations"

Partial Mocks

Partial mocking is a pretty useful way to ensure your methods are tested independently from each other, and while it is supported by all of the libraries tested here, some make it much easier to work with than others.
Gist "Partial mocks"

Chaining Attributes and Methods

I'm of the opinion that chained attributes are generally indicative of poor separation of concerns, so I don't place too much weight on how the different libraries handle them. That said, I've certainly had need of this functionality when dealing with a settings tree, where it can be much easier to just create a mock if you need to access settings.a.b.c.

Chained methods are sometimes useful (especially if you use SQLAlchemy), as long as they don't impair readability.
Gist "Chaining methods and attributes"

Failures

An important part of any testing tool is how informative it is when things break down. I'm talking about detail of error messages, tracability, etc. There's a couple of errors I can think of that are pretty common. For brevity, I'm only going to show the actual error message, not the entire traceback.

Note: Mock is a bit of an odd duck in these cases, because it lets you do literally anything with a mock. It does have assertions you can use afterwards for most cases, but if an unexpected call is made on your mock, you will not receive any errors. There's probably a way around this.

Arguments don't match expectations, such as when we call time.sleep(4) when our expectation was set up for 6 seconds:

Mocker: MatchError: [Mocker] Unexpected expression: m_time.sleep(4)
Flexmock: InvalidMethodSignature: sleep(4)
Fudge: AssertionError: fake:time.sleep(6) was called unexpectedly with args (4)
Mock: AssertionError: Expected call: sleep(6)
Actual call: sleep(4)

When I first encountered Flexmock's InvalidMethodSignature, it threw me off. I think it could certainly be expanded upon. Otherwise, Mock and Fudge have very nice messages, and as long as you know what was supposed to happen, Mockers is perfectly sufficient.

Unexpected method called, such as when you misspell "sleep":

Mocker: MatchError: [Mocker] Unexpected expression: m_time.sloop
Flexmock: AttributeError: 'Mock' object has no attribute 'sloop'
Fudge (patched time.sleep): AttributeError: 'module' object has no attribute 'sloop'
Fudge: AttributeError: fake:unnamed object does not allow call or attribute 'sloop' (maybe you want Fake.is_a_stub() ?)
Mock: AssertionError: Expected call: sleep(6)
Not called

Mock doesn't tell you that an unexpected method was called. Mocker has what I consider the best implementation here, because it names the mock the call was made on. The second Fudge variant is good, but because you might encounter it or the first variant depending on context, Fudge overall is my least favourite for this. Flexmock simply defers handling this to Python.

Expected method not called:

Mocker: AssertionError: [Mocker] Unmet expectations:
=> m_time.sleep(6)
- Performed fewer times than expected.
Flexmock: MethodNotCalled: sleep(6) expected to be called 1 times, called 0 times
Fudge: AssertionError: fake:time.sleep(6) was not called
Mock: AssertionError: Expected call: sleep(6)
Not called

I think they all do pretty well for this case, which is good, because it's probably the most fundamental.

Roundup

So, having spent a bit of time with all of these libraries, how do I feel about them? Let's bullet point it!

Mocker

Pros

Very explicit syntax
Verbose error messages
Very flexible

Cons

Doesn't support Python 3 and not under active development
Performance sometimes isn't very good, especially with patch()
Quite verbose

Flexmock

Pros

Clean, readable syntax for most operations

Cons

Syntax for chained methods can be very complex
Error messages could be improved

Fudge

Pros

Using @patch is really nice, syntactically
Examples showing web app testing is nice touch

Cons

@patch can interfere with test runner operations (because it affects the entire interpreter?)
Partial mocking is difficult

Mock (preliminary)

Pros

Very flexible

Cons

Almost too flexible. All-accepting mocks make it easy to think you have better coverage then you do (so use coverage.py!)

Acknowledgements

Clearly, a lot of work has been put into these mock libraries and others. So I would like extend some thanks:

Gustavo Niemeyer, for his work on Mocker.
Kumar MacMillan, for his work on Fudge, and for helping me in preparing material for this post.
Herman Sheremetyev, for his work on Flexmock
Michael Foord, for his work on Mock, and for getting me on Planet Python

Additionally, while I didn't work with them for this post, there are a number of other mock libraries worth looking at:

Dingus
Mox
MiniMock, which I've used quite a bit in the past, and I'm delighted to learn that development is continuing on it!

Blogging and not blogging

2011-09-07T15:31:00.000-07:00

It's been a month since I started working on this blog, and the only thing I have gotten up is my introductory post, which talks about all the other stuff I want to write. Not quite the start I wanted to get off to, admittedly, but August managed to be much busier than I would have thought, me lacking a job and all.

That said, I do have a bone to pick with Blogger. It's convenient, free, interfaces with all my stuff, but writing things that include code has been a real pain in the butt. I could inline it as a quote, but I don't get syntax highlighting. It seems I can modify my template to include plugins for syntax highlighting, but given that I would like these bits to be copy-pastable, including the actual code in the HTML seems like a recipe for trouble.

So I am currently using "gist" for my samples, and embedding them with script tags. This is nifty, certainly, but I'm guessing it doesn't come across well in RSS, and probably to aggregators like Planet Python (which is probably where you are reading this). Furthermore, if I'm already maintaining a github account to get the gists, and for the larger project work, it seems like it would be much more straightforward if I could embed snippets directly from github, so I could do my editing and testing in PyDev.

I'm sure I'm not the first person to run up against this, so we will see if my Planet Python syndication can bring any knowledgeable commentators my way. If you have any suggestions, especially ones where you can link to examples, that would be very much appreciated.

Introductions

2011-07-29T10:21:00.000-07:00

Hi there, I'm Alec Munro. For various reasons, I've been unable to engage online for the last several years. That is no longer true, and I find myself with time, so here I am, engaging (well, I hope so).

So what am I trying to engage with? Well, I'm a software developer, interested in refining the process of software development to produce higher quality software. In practice, my skills and experience tend towards Python and Web Development, with the last 5 years being spent in the realm of test automation. Let's break that down...

When I say I want to produce higher quality software, I'm defining quality as "that which meets or exceeds my customer's expectations". This is a definition I'm stealing from someone, but which feels correct to me. There are two implicit conditions in that definition that go beyond simply writing good code: that you know who your customers are, and that you understand their expectations. So for me, developing quality software is a very holistic process, that encompasses the entire lifecycle of a desired piece of functionality.

I've been developing web sites for 14 years, starting with static HTML and JavaScript, moving through Flash and PHP, and finally "settling" on Python about 8 years ago. Since then, web development has almost never been my primary occupation, but it has almost always been a tool I could use to improve the projects I was working on. In the Python world, I started with Zope 2, moved to Zope 3, then Grok, and most recently Pylons and Pyramid. I'm biased towards building everything as a REST-ish web service, with a UI built in JQuery.

I came to test automation because I saw a need. I saw (seemingly) automatable tasks being performed by manual testers, and I took action to address this. I learned, with great difficulty, that many things I had assumed could be easily automated were actually terrible candidates for automation, often due to the development process that produced them. I learned about "testability" (thanks Misko!), and the role that test automation must play in driving development process improvements. I also gained an appreciation for what I call "proactive transparency", which is the philosophy that much of the information people need is too difficult to find, because those producing it don't know how to publicize it (or don't appreciate the need to do so). In test automation, this applies to results of testing, but I've found it applicable to many other areas.

So that's a rough overview of who I am, and what I can do. But where's the engagement? Well, I find myself without a job, but with a bit of a margin before I need a job. So I thought I would take this time to write on subjects I know about, as well as to study and report on those that I don't. This will hopefully give me the opportunity to broaden my skills (specifically for interviewing purposes, I will admit), while also re-engaging with the development community that I have been absent from for several years. For you, the reader, hopefully some of these topics will interest you, and you will participate in my discussions, to correct me where I err, or weasel further answers out of me when I am unclear.

What kind of topics? Well, I have no formal computer science training, so there's a couple of areas that tend to come up in interviews that I don't think I give satisfactory answers on. These include:

Design patterns. I know many of their names, and I know I use them often in my work, albeit uncredited. I will research the most common ones, and invent problems where they are applicable, and explain why one over the other.
Sorting. Quick sort, merge sort, etc. Using Python for web development, I've never been held back by the performance of a sorting operation, but I understand that these algorithms are considered quite foundational, and I know I've suffered in interviews due to the lack of them.

I will also go into some topics that are interesting to me, and play to the strengths I have developed during my career.

Mock library comparison. I consider the use of mock objects to be an essential (and often neglected) part of almost any software development project. I also think there are too many Python mock object libraries. :) I will discuss this in more detail, and show examples of the use of each library, and try to illustrate the strengths and weaknesses of each.
Web application development. Because of my recent work history, I have virtually no public code to share with potential interviewers. So, purely as a vanity project, I will create what I consider to be a properly tested and documented web application, and write about the process. I'll try to make a useful application, but I can't promise that.
RabbitMQ, ZeroMQ, Gevent. I keep running across these things, but I've never had an occasion to do much beyond skim their docs. They seem interesting and well regarded by those who are well regarded by me, so I'll look into them.
Go (the language, I'll leave the game to my father, at least for now). I've heard being laid off is a great time to learn a new language, and two of the jobs I've seen that would be most interesting mention Go, so I think that will probably be it for me.

To further my ideal of engagement, I will post any code I write for any of these in github (also new to me), and encourage contributions from the community. Hopefully, the community will find enough useful in what I create to do so.