More Node.JS Module Patterns

The post on Node.JS module patterns and the slideshow from the talk I did at OttawaJS keep getting a lot of mentions. Those are really simple examples, but in practice most modules have a bit more substance to them.

There are a couple of patterns that I've seen used in Express.JS apps a lot.  One simply exports a bunch of functions that are used as route handlers. Another passes an object to the module, and the module attaches things to it.  And the last one exports an Express router object that the main app can use to define more specific routes on a base URL.

Let's look at these with some simple examples.

Exporting route handler functions

This is a fairly common pattern where the module simply exports a number of functions. In this case they're route handler functions that an Express app can use to handle the various routes it declares.

users.js

exports.getUser = function (req, res, next) {
  res.send('respond with a user');
};

exports.updateUser = function (req, res, next) {
  res.send('update user and respond');
};

app.js

var users = require('users.js');

app.get('/user/:userid', users.getUser);
app.put('/user/:userid', users.updateUser);

Passing in and enhancing an object

This pattern has been used in some Express example apps and shows how you can pass variables into a module, either to use them in the module or to "enhance" an object by attaching things onto it. Of course you have to know what you're doing and not have two different modules that try to do the same thing.

users.js

module.exports = function (app) {
  app.get('/user/:userid', function (req, res, next) {
    res.send('respond with a user');
  });
};

app.js

var users = require('users.js')(app);

Express router modules

The most recent version of express-generator creates a sample app using this pattern. Route modules are created under the routes/ directory, and they require an instance of the Express router. They add their routes to the router instance, and set module.exports as the router object. The main app.js file then attaches these router objects via app.use('/path', router) as you can see below. It's a nice clean way to organize route modules in Express.

users.js

var express = require('express');
var router = express.Router();

router.get('/:userid', function (req, res, next) {
  res.send('respond with a user');
});

module.exports = router;

app.js

var express = require('express');
var users = require('users.js');
var app = express();
app.use('/users', users);

Fluent Talk Summary: Brendan Eich, "JavaScript: the High and Low Roads"

In this lighthearted talk, Brendan Eich (inventor of JavaScript) discusses high and low-level improvements in upcoming versions of JavaScript. Acknowledging that Web development is hard ("like having a chain-saw in place of a hand"), Eich says the upcoming version of JavaScript, ES6 "Harmony", will address many difficulties facing web developers, with improvements like a built-in module system, observable objects, and many other features that will make "transpilers" and other "syntactic sugar" libraries unnecessary.

The future ES7 version of JavaScript aims to bring many low-level improvements for high-performance and scientific computing, including new value objects, operator overloading, and SIMD intrinsics. New value objects proposed in ES7 include: int64, uint64, int32x4 and int32x8 (SIMD), float32 (useful for GPUs), float32x4 and float32x8 (SIMD), bignum, decimal, rational, and complex. ES7 will also introduce operator overloading and new literals like 0L (int64), 0UL (uint64), etc. Support for SIMD (Single Instruction, Multiple Data) will lead to native performance for applications like game programming and signal processing. To demonstrate further, Eich reveals the first-ever public demo of the Unreal Engine 4 from Epic Games, showing stunning 3D graphics running at a full 60 frames per second, with no plugins, in a Firefox build with special SIMD capabilities.

Guided by the Extensible Web Manifesto, the high road of developer-friendly features and the low road of safe, low-level language improvements will converge in a virtual machine that offers native performance while being very developer friendly.

In a nutshell - "Always bet on JS!"

My rating: 4/5. Informative, insightful and entertaining for programmers.

Dynamic Types FTW

There's a belief among some programmers, particularly the OO classical inheritance folks, that static typing is a bulwark of security against all kinds of disasters that may happen in your code.  They feel that the compiler will ensure their programs work correctly, by generating errors and warnings and refusing to compile their program until all the mistakes are fixed.

Some developers mistakenly speak in terms of "strong typing" and "weak typing". Put that way, who wouldn't prefer "strong" over "weak" typing? In reality the terms "strong typing" and "weak typing" are undefined in the language of computer science. They're misnomers that people sometimes use to talk about strict (explicit) type definition with static type checking, versus dynamic (runtime) type checking with implicit type conversion.

It's unfortunate that this line of reasoning causes some people to avoid languages that offer dynamic types. Dynamic types can be very useful, and static typing actually offers very little in the way of ensuring program correctness.  I finally sat down and watched Eric Elliott's presentation "Static Types are Overrated: The Dynamic Duo - Loose Types and Object Extension" and was treated to a bang-on description why static types aren't all they're cracked up to be.  Eric does a good job of explaining and clarifying things about JavaScript and programming in general that I intuitively believed but had a difficult time putting into words or backing up with solid examples.

I borrowed a bunch of points from Eric's talk when I spoke at a recent Ottawa JavaScript meetup. I was presenting on Flow, a static type checker for JavaScript, but I wanted people to see it as something that could be added to their regular code linting process rather than something to enforce a statically typed programming style. Here are some of them.

• It's a myth that dynamic loosely typed languages like JavaScript aren't suitable for building large, complex applications.  There are dozens of examples from Facebook to 37signals, Dow Jones, Adobe, Flickr, LinkedIn, Walmart, etc).
• Type correctness does not guarantee program correctness. This is one of the biggest cargo-cult myths out there: thinking that static, strict type checking will save you. It won't.
• When languages lack dynamic types, people fake it with ugly hacks - void pointers, variadic functions, abusing array accessors, type-casting of all sorts, generics and ugly template classes, to name a few. I don't know any C or Java programmers who haven't used and abused at least a few of these techniques.
• "Any sufficiently advanced C / Fortran [Java, etc] program contains an ad hoc, informally specified, bug-ridden, slow implementation of half of common Lisp" - Greenspun's Tenth Rule. (There are no rules 1 through 9, by the way. Greenspun's Rules of Programming start and end with number 10.)
• Older statically typed languages are now introducing dynamic types because they really are useful. Objective-C added them a long time ago; more recently C++11 has added dynamic types, Java has "generics", and libraries like cppscript and boost provided ways to do dynamic types in earlier versions of C++.
• Functional programming benefits from functions that can operate on any type which implements its requirements (for example, arguments have a valueOf method). Otherwise known as duck typing. This permits things like map, filter, forEach etc. to be used generically.
• Correctness can only really be assured through proper unit and integration testing. 
• Code that is well organized in small, simple modules, linted, unit tested, peer reviewed, and integration tested, is very unlikely to contain type errors. 

The last thing I would add is that while static types are over-rated, static code analysis tools like JSLint, Tern.js and Flow are mostly underrated and underutilized. If you use something like Flow or Tern.js to provide hints and insights into potential type mismatches, the chances of type errors become essentially zero. Ideally they should be as up-front as possible, i.e. integrated directly into your code editor. I wrote a JSLint plugin for SublimeText and recently wrote a Flow plugin as well. They provide immediate static code analysis warnings as I edit my code, so I can fix problems before I commit anything to the source code repository.

Three Simple Rules for Escaping Callback Hell

A lot of newcomers to Node.JS complain about "callback hell" and the "pyramid of doom" when they're getting started with the callback-driven continuation passing style.  It's confusing, and a lot of people reach for an async / flow-control module right away.  Many people have settled on using Promises, a solution that brings some unfortunate problems along with it (performance, error-hiding anti-patterns, and illusory behavior, for example).

I prefer using some simple best practices for working with callbacks to keep my code clean and organized. These techniques don't require adding any extra modules to your code base, won't slow your program down, don't introduce error-hiding anti-patterns, and don't convey a false impression of synchronous execution. Best of all, they result in code that is actually more readable and concise, and once you see how simple they are, you might want to use them, too.

Here they are:

1. use named functions for callbacks
2. nest functions when you need to capture (enclose) variable scope
3. use return when invoking the callback

The Pyramid of Doom

Here's a contrived example that uses typical node.js callbacks with (err, result) arguments. It's a mess of nested functions: the so-called Pyramid of Doom. It keeps indenting, layer upon smothering layer, until it unwinds in a great cascading spasm of parenthesis, braces and semi-colons.

Named Callbacks

The Pyramid of Doom is often shown as a reason to use Promises, but most async libraries -- including and especially Promises -- don't really solve this nesting problem.  We don't end up with deeply nested code like this because something is wrong with JavaScript. We get it because people write bad, messy code.  Named callbacks solve this problem, very simply. Andrew Kelley wrote about this on his blog a while ago ("JavaScript Callbacks are Pretty Okay"). It's a great post with some simple ways of taming "callback hell" that get skipped over by a lot of node newcomers.

Here's the above example re-written using named callback functions. Instead of a Russian doll of anonymous functions, every function that takes a callback is passed the name of the callback function to use. The callback function is defined immediately afterwards, greatly improving readability.

Nest Only for Scope

We can do even better. Notice that two functions, sendGreeting and showResult, are still nested inside of the getGreeting function. Nested "inner" functions create a closure that encloses the callback function's own local variable scope, plus the variable scope of the function its nested inside of. These nested callbacks can access variables from higher up the call stack. In our example, both sendGreeting and showResult use variables that were created earlier in the getGreeting function. They can access these variable from getGreeting, because they're nested inside getGreeting and thus, enclose its variable scope.

A lot of times this is totally unnecessary. You only need to nest functions if you need to refer to variables in the scope of the caller from within the callback function. Otherwise, simply put named functions on the same level as the caller. In our example, variables can be shared by moving them to the top-level scope of the greet function. Then, we can put all our named functions on the same level. No more nesting and indentation!

Return when invoking a Callback

The last point to improve readability is more a stylistic preference, but if you make a habit of always returning from an error-handling clause, you can further minimize your code. In direct-style programming where function calls are meant to return a value, common wisdom says that returning from an if clause like this is bad practice that can lead to errors.  With continuation-passing style, however, explicitly returning when you invoke the callback ensures that you don't accidentally execute additional code in the calling function after the callback has been invoked. For that reason, many node developers consider it best practice. In trivial functions, it can improve readability by eliminating the else clause, and it is used by a number of popular JavaScript modules.  I find a pragmatic approach is to return from error handling clauses or other conditional if/else clauses, but sometimes leave off the explicit return on the last line in the function, in the interest of less code and better readability. Here's the updated example:

Compare this example with the Pyramid of Doom at the beginning of the post. I think you'll agree that these simple rules result in cleaner, more readable code and provide a great escape from the Callback Hell we started out with.

Good luck and have fun!

doxli - a help utility for node modules on command line

Quite often I fire up the node REPL and pull in some modules I've written to use on the command line. Unfortunately I often forget the exact way to call the various functions in those modules (there are a lot) and end up doing something like foo.dosomething.toString() to see the source code and recall the function signature.

In the interest of making code as "self-documenting" as possible,  I wrote a small utility that uses dox to provide help for modules on the command line. It adds a help() function to a module's exported methods so you can get the dox / jsdoc comments for the function on the command line.

So now foo.dosomething.help() will return the description, parameters, examples and so on for the method based on the documentation in the comments.

It's still a bit of a work in progress, but it works nicely - provided you actually document your modules with jsdoc-style comments.

All the info is here: https://www.npmjs.org/package/doxli

Repackaging node modules for local install with npm

If you need to install an npm package for nodejs from local files, because you can't or prefer not to download everything from the  npmjs.org repo, or you don't even have a network connection, then you can't just get an npm package tarball and do `npm install <tarball>`, because it will immediately try to download all it's dependencies from the repo.

There are some existing tools and resources you can try:

• npmbox - https://github.com/arei/npmbox
• https://github.com/mikefrey/node-pac
• bundle.js gist -  https://gist.github.com/jackgill/7687308
• relevant npm issue - https://github.com/npm/npm/issues/4210

I found all of these a bit over-wrought for my taste. So if you prefer a simple DIY approach, you can simply edit the module's package.json file, and copy all of its dependencies over to the "bundledDependencies" array, and then run npm pack to build a new tarball that includes all the dependencies bundled inside.

Using `forever` as an example:

1. make a directory and run `npm init; npm install forever` inside of it
2. cd into the node_modules/forever directory
3. edit the package.json file
4. look for the dependencies property
5. add a bundledDependencies property that's an array
6. copy the names of all the dependency modules into the bundledDependencies array
7. save the package.json file
8. now run `npm pack`. It will produce a forever-<version>.tgz file that has all it's dependencies bundled in.

Update: another proposal from the github thread (I haven't verified this yet):

1. In online environment, npm install --no-bin-link. You will have a entire flattened node_modules
2. Then, bundle this flawless node_modules with tar / zip / rar / 7z etc
3. In offline environment, extract the bundle, that's it

Node.JS Module Patterns using simple examples

Slides for a recent talk at Ottawa.JS on "Node.JS Module Patterns using simple examples" are available. The slides have been updated to include a brief intro to Common.JS, examples for exporting named and anonymous functions, objects and prototypes, and an explanation of "exports" vs. "module.exports".

JavaScript Efficiencies

I'm working on an article about patterns for structuring Express.JS apps, which is taking too long, so I decided to write this instead: Here are a few tips and tricks for JavaScript programming that I like.

Comment switches

Comment switches let you comment out entire blocks of code with a single character, or switch between two different blocks of code with a single character, which can be useful when prototyping.

//* 
console.log("Hello!\n"); 
/*/  
console.log("Goodbye!\n"); 
// */

Removing the first slash '/' toggles between these two print statements. See the original post for more examples of comment switches.

Iterate by Counting Down

You can iterate n times concisely, like this:

var n = 1000;
while (n--) { ... }

Defaulting Arguments

This is a handy way to provide a default value for undefined arguments in a JavaScript function.

function foo(bar) {
   var bar = bar || "Some default";
   ...
}

Testing Express.JS REST APIs with Mocha

My post on Testing REST APIs with RESTClient and Selenium IDE has been getting some attention, but you might also want an automated testing solution to be part of your build process.

This post will cover testing REST API's built with Express.JS, using Mocha as the test framework. Express and Mocha are both by the same author and work well together.  This technique can also be used to test Express apps in general.

In a nutshell, we'll create a test suite in Mocha that does the following:

1. instantiates and runs our REST API server (an Express app)
2. sends HTTP (or HTTPS) requests to the REST API
3. checks the responses for appropriate values

So, let's say you've written an Express app that works as a REST API server, and now you want to automatically test it over HTTP. Assuming you have Mocha installed, and are somewhat familiar with how to use it, building a suite to test your Express app over HTTP is relatively simple.

Our example test suite is a file located at test/app_test.js. The first thing you'll need to do is include the Express app you want to test. Mine is simply called app.js, so this is what the start of my test suite looks like:



Before we can start testing our API server app we need to start it. We can create a test suite called "app" for this. Before anything, we'll start the app. Then we'll check to make sure it is running. Here's what it looks like:



There are a couple of pieces missing from the test case above. We're using Node's build-in http library, so we need to include that at the top of our file as well:



And, there's a function defaultGetOptions, which is just a little helper utility to format our request:



Lastly, you can see that there's a variable called sessionCookie. This is how we maintain a session with our REST API, in case your API requires an authenticated session. You can simply declare this variable at the top of the file:



And set the cookie value whenever you get a response from the API server. Here is an example of setting the sessionCookie value when logging in:



As you can see, I'm using a few extra variables or functions - there are some test-user parameters, and a function to format the headers for a POST request. Having seen the example for a GET request, you should have no problem creating similar functions for POST, PUT, and DELETE (check out the Node docs for HTTP).

Finally, to make this all work you might need to modify your Express app to make sure that it does not start up automatically on the default port, if it is being included as part of a test suite. You do this by checking for module.parent.

Above, you can see that the app will only start actively listening if it is being run directly, but if it has been included in a test suite like ours, it won't automatically start up. This let's us start the app on a port of our choosing from within our test suite, like we did in the example above.

Hope that helps...

Machine Learning in JavaScript

I did a presentation at Ottawa JavaScript on machine learning, which covered a lot of the material in my two recent posts on Bayesian classifiers. This was new to a lot of the audience, so I made slides to step through a very simple example.

Using the "box of chocolates" analogy, the slides demonstrate how to predict if a chocolate contains nuts, depending on its colour and shape.

You can view the slides here:

http://darrenderidder.github.io/talks/MachineLearning

Using local npm executables

When developing NodeJS apps, it used to be common practice to install module dependencies in a central node_modules folder. Now, it's recommended to install dependencies for each project separately. This allows fine-grained control over specific modules and versions needed for each application and simplifies installation of your package when you're ready to ship it.  It does require more disk space, but these modules are mostly very small.

For example if your project had this package.json file, running npm -install would create a node_modules sub-directory and install express and mocha into it:

{

  "name": "awesome-app",

  "version": "0.0.1",

  "dependencies": {

    "express": "3.0.0rc3"

   ,"mocha": "latest"

  }

}

It's also fairly common to install some modules globally, using npm -g, if they provide command-line utilities. For example, if you wanted to have jslint available to all users on your system, you'd do this:

sudo npm -g install jslint

which would place this in /usr/local/bin/jslint.

If you specify a dependency in your package.json file, and that module includes an executable -- like jslint, for example, or mocha -- then npm install will automatically place those executables in your project's node_modules/.bin directory. To make use of these, add this prefix to your $PATH in your ~/.bashrc file:

export PATH=./node_modules/.bin:$PATH

This allows each one of your NodeJS projects to independently specify, install and use not only packages that are required in your JavaScript code, but also the command-line tools you need for testing, linting, or otherwise building your app.

Serve gzipped files with Express.JS

Assuming you have minified, gzipped versions of your static javascript files already available (i.e. jquery.min.js.gz) you can serve the compressed files relatively easily. You don't really need to use on-the-fly gzip compression. The obvious benefit is faster page loading.  Although it might seem like a small thing, saving a few kilobytes here and there in the midst of MB-sized downloads, such small performance optimizations can and *do* have a noticeable impact on responsiveness and overall user experience... just ask Thomas Fuchs.

If you had, for example, myscript.js, you could simply do this:

uglifyjs -nc myscript.js > myscript.min.js

cat myscript.min.js | gzip > myscript.min.js.gz

And then with a simple bit of URL re-writing, serve the compressed version whenever a supported browser requests the minified version:

Express 2.x :

// basic URL rewrite to serve gzipped versions of *.min.js files
app.get('*.min.js', function (req, res, next) {
  req.url = req.url + '.gz';
  res.header('Content-Encoding', 'gzip');
  next();
});

Express 3.x :

app.get('*.min.js', function (req, res, next) {
  req.url = req.url + '.gz';
  res.set('Content-Encoding', 'gzip');
  next();
});

Now, if the browser requests "./js/myscript.min.js", it will actually get a copy of myscript.min.js.gz from that URL, without doing a redirect, and the browser will automatically decompress the file and use it in the normal way.

If your scripts are small to start with, the benefits won't be that great. Concatenating all your scripts together, then compressing them, could offer a major improvement over several smaller files.

Note that this doesn't check to see which files actually have gzipped versions available, so you would need to be diligent about keeping your files current, or write a simple function to cache a list of the available gzipped files at run time. 

JavaScript Closures for 6-year-olds

When I first became interested in NodeJS I decided I needed to understand closures. They're supposed to be an advanced topic in JavaScript, but they're kind of fundamental to Node's asynchronous, callback driven continuation-passing style.

My background is mainly in C/C++, so unlike Ruby programmers, who have probably been exposed to closures, I had to study up on it.

After reading some excellent technical descriptions and layman's articles, I came across a Stack Overflow post asking "How do JavaScript Closures Work?", which is really good. The writer was asking how you might explain closures to a six-year-old, because Richard Feinman said that "If you can't explain it to a six-year-old, you don't really understand it yourself".

Here's my attempt at an explanation of closures for six-year-olds:

A man goes on a long trip, leaving his home and family behind. He's very fond of his home and before he leaves he takes a picture to bring with him. On his travels he is sometimes asked to describe his home. At such times, he takes out the photograph and begins to describe it. He talks about the white picket fence, the flower boxes by the windows, and the apple trees in the back yard.

While he's away, however, his family has decided to renovate and build a 3-story condo on the spot where the house is. Gone is the white picket fence. Gone are the apple trees. Gone are the flower boxes by the windows. The entire scene has changed. Yet the travelling man knows nothing about these changes, and continues to talk about his home as he remembers it, pulling out his worn photograph to show to strangers along the way. He describes the white picket fence, the flower boxes and the trees. Unaware of the changes being made while he is away, in his mind and in his pocket he carries a picture of home that is a closure - a snapshot of the way things were - and that is all he knows.

In JavaScript, when you return a function, its like a traveling man. The function will remember the scene where it came from, just the way it was.

var color = 'white';

function leaveHome () {
   // a variable in leaveHome's scope

   var description = "My house is " + color;
   // returns another function, creating a closure over description

   return function() {

      console.log(description);

   }

}

var travellingMan = leaveHome();

travellingMan(); // "My house is white"

color = 'red';   // They painted the house

travellingMan(); // "My house is white"

A couple points to notice: 

1. In JavaScript, functions are objects, too. Functions can create and return other functions.
2. When you return a function or pass one as an argument, it doesn't need a name. It can be defined right in-line as an anonymous object.
3. When the return function is created it "closes over" the variables in it's scope. In the example it takes a snapshot of the description variable.

Simple Intro to Synchronization / Serialization in NodeJS

One of the most common topics that comes up with people who are new to NodeJS is how perform synchronous tasks in an asynchronous programming environment. There are two general responses to this:

1. Your Doing It Wrong. Just learn how to deal with the asynchronous programming style.
2. Use a module. Step.js, asynch.js, or any number of other ones.

A lot of people have tackled this problem. You might even hear someone recommend to go write your own flow control library first, then go choose the module that you like best. To be honest, I've always wanted to understand the internals of control flow libraries like Step.JS, and have even pored over over Isaac Schlueter's Slide.JS slideshow.

Recently I wanted to create a little node script that would populate some tables in a database. I had a bunch of methods to do this, but they were all asynchronous functions and I needed to call them serially. For example, the first one would create a table. The second would populate some data in the table. And so on. Since my use case was fairly simple, and I didn't really want to download and start blindly using a flow control library to solve my problem, I decided to take it as an opportunity to see if I could create my own solution for serializing the function calls.

The resulting code is below. It's not especially pretty, or flexible. But after working it out, I felt that I did understand the general approach to serialization / synchronization in NodeJS a lot better. Hopefully looking at this simple example will shed some light on this for you as well.

To start with, I imported my init module. The functions I wanted to call synchronously are all exported by this module. I put these functions in an Array like this (note that I'm not actually invoking the functions here, just putting references to them in the array):



The next thing I did was create an array called "callbacks":



The idea is to create a bunch of special callbacks that will be invoked after each of the "steps" above. A callback function normally has parameters like (err, result), and a body that handles these. In addition to handling the err and result, however, when my callbacks were invoked I needed my callbacks to call the next step in the list. You can picture it something like this:

+--------+          +------+
| func_1 |--------->| cb_1 |
+--------+          +------+
                       |
+--------+             |
| func_2 |<------------+
+--------+
    |               +------+
    +-------------->| cb_2 |
                    +------+
           etc.

Ok, bad ascii art, but hopefully you get the idea. Function 1 gets called, finishes, and callback 1 is invoked. Callback 1 does the usual err / result handling and then invokes Function 2. Function 2 finishes and callback 2 is invoked. It does the usual err / result handling and then invokes Function 3, and so on.

For each of my "steps", I need to create one of these smart callbacks that knows the next function to call. I do that in a for loop, creating a custom callback for each function in the list of "steps".



Once the customized callbacks are all created, I start off the chain of events by calling the first function and passing in the first custom callback, like tipping the first domino in a row.



The part where we create the custom callbacks is the most interesting. Its like a little factory that creates callback functions. The createCallback function creates custom callbacks and returns them. The idea of a function returning another function is a little odd, if you're new to functional programming. But in JavaScript, a function can be passed around like any other variable. And here we are creating custom callback functions that know the next function in the list of steps.  They also know that - if there is a next function to call - they should invoke it by passing in the next callback.



Obviously this is a super simplified example, and doesn't even come close to the functionality provided by the flow control libraries that are readily available. But hopefully its simple enough to give you a basic idea of how some of these modules go about solving the problem of taking asynchronous functions, and executing them in a synchronous, or serialized way.

For reference, here's the whole file... good luck and have fun!

A Simple Introduction to Behavior Driven Development in NodeJS with Mocha

I recently had a chance to use the Mocha test framework for NodeJS on a large-ish project and wanted to provide a really simple introduction to it.  Mocha is a unit testing framework for NodeJS apps from TJ Holowaychuk.  As the successor to the popular Expresso test framework, it adds a number of new features. It supports both Test-Driven and Behavior-Driven development styles and can report test results in a variety of formats.

Mocha can be installed easily using npm. Assuming you have node and npm installed, you can install mocha as a global module using the -g flag to it will be available everywhere on the system.

npm -g install mocha

or perhaps

sudo npm -g install mocha

or, if you prefer a local install instead of a global install

npm install mocha
export PATH=./node_modules/.bin:$PATH

You're also going to want should.js, which you can install in your local project folder.

npm install should

Mocha, in addition to being a test framework, is a command-line utility that you run when you want to test things. You should be able to type `mocha -h` on the command line now and get some usage info.

Example

Before we can use mocha to test stuff we should have a little bit of code to test. I'm going to borrow from Attila Domokos' blog http://www.adomokos.com/2012/01/javascript-testing-with-mocha.html but try to simplify it just a bit. We're going to create two files in two different directories:

src/dog.js

test/dog_test.js

dog.js is our source code that we want to test.  Normally this would be a module that you have written. Since this is a super-simple introduction, it just exports one function called "speak". We can use this to invoke the "speak" method, which will return "woof!".

// dog.js
function speak() {
    return 'woof!';
}
exports.speak = speak;

Now we can write something to test our code: dog_test.js. Remember this file goes under the test/ directory and dog.js is under the src/ directory. So here is how we would write a test case for Dog.

// dog_test.js

var should = require('should'); 
var dog = require(__dirname + "/../src/dog");

describe('dog', function() {
  it('should be able to speak', function() {
    var doggysound = dog.speak();
    doggysound.should.equal('woof!');
  });
});

Now there should be a src/ and a test/ directory with these two files in them.

src/dog.js
test/dog_test.js


Mocha will automatically run every .js file under the test/ directory. To run the tests you can simple type 'mocha' on the command line:

mocha

and it should give you the following output:

  ✔ 1 test complete (3ms)

or if you want a little more detail use:

mocha -R list

and it should give you this output:

  ✓ Dog should be able to speak: 0ms
  ✔ 1 test complete (2ms)

Mocha can generate several different styles of test reports. To see a list of the types of reports it can generate, type mocha --reporters.

Explanation

Let's take a quick look at the test case definition. A group of tests starts with "describe". You provide some title for the thing you're testing. You can choose whatever title you want, but it should be descriptive. Our test suite is called 'dog'.

describe("dog", function () {
    // your test cases go in here
}

You can describe more than one group of tests in the file.

describe("dog", function () {
    // your doggy tests
}

describe("puppy", function () {
    // your puppy tests
}

Test descriptions can be nested, which is often convenient.

describe("dog", function () {
    describe("old tricks", function () {
        //  your old doggy tricks
    }
    describe("new tricks", function () {
        // your new doggy tricks
    }
}

Individual tests themselves start out like "it('should do such-and-so...". You can give the test whatever description you want, but generally you give it a brief explanation of what it should or shouldn't do. If you added more methods to Dog, you would add more tests within the "Dog" test description to test them.

describe("dog", function () {
    it('should be able to speak', function () {
        // the body of the speak test goes here
    }
    it('should be able to wag its tail', function () {
        // the body of the wag test goes here
    }
}

Our examples use should.js for writing test cases in Behavior Driven Development style. Should.js provides a bunch of cool assertions like foo.should.be.above(10) or baz.should.include('bar'), and so on. Check out the documentation for should.js here: https://github.com/visionmedia/should.js

    it('should be able to speak', function () {
        var doggysound = dog.speak();
        doggysound.should.equal('woof!');
    }
    it('should be able to wag its tail', function () {
        var wags = dog.wag(5);
        wags.should.be.above(4)
    }

Synch and Asynch Tests

These examples show an asynchronous testing approach. It's quite easy to run your tests in a synchronous way, though, by using the "done" callback which Mocha provides.


    it('should be able to speak', function (done) {
        var doggysound = dog.speak();
        doggysound.should.equal('woof!');
        done();
    }


Using the "done" callback means the test will complete before moving on to the next one.

For a slightly more advanced tutorial check out http://dailyjs.com/2011/12/08/mocha/. It has some great tips about setting up a package.json and Makefile to help with test automation, as well as writing synchronous and asynchronous tests.

A simple introduction to TDD with Expresso in NodeJS

In the last post we made a super simple NodeJS module called hello.js and used it in app.js. We can make it better, and test it with Expresso.

First we can add to the awesomeness of hello.js by adding a farewell function, so it can say both "hello" and "goodbye".

// hello.js
var greeting="Hello World!";
var farewell="Goodbye Cruel World!";
var greet = function(){
    return greeting;
};
var leave = function() {
    return farewell;
};
exports.greet = greet;
exports.leave = leave;

So now our module can say hello, and bid farewell.  It exports the "greet" and "leave" functions, which we can use from app.js like so:

// app.js
var hello=require('./lib/hello.js'); 
console.log(hello.greet());
console.log(hello.leave());

Notice that I moved hello.js into a new directory called "lib", which is why the require statement now says "require('./lib/hello.js')".

Now suppose we want to test this module using expresso.  Since expresso is a command-line utility, we should install it globally:

npm install -g expresso

Now we can write a test.  First we make a "test" directory and then create a file called "hello.test.js".

mkdir test
vi test/hello.test.js

// test/hello.test.js
var assert = require('assert'),
      hello = require('hello.js);

module.exports = {
    'greet()': function() {
        assert.equal('Hello World!', hello.greet());
    }
};

This just requires node's built-in "assert" module for unit testing, plus our hello.js module that we want to test. Then, it defines a test to make sure that when we call "greet()", the response is "Hello World!". Alright! Now we can go back to the parent directory (where app.js lives) and run expresso. We tell it to look in the "lib" directory for hello.js. Expresso already knows to look in the "test" directory for tests, and will run everything it finds there.

$ expresso -I lib

    %100 1 tests

Wonderful.  Only, we didn't test the part where we say goodbye. Expresso has built-in code coverage reporting that we can enable with the "-c" flag:

$ expresso -I lib -c

   Test Coverage

   +------------------------------------------+----------+------+------+--------+
   | filename                                 | coverage | LOC  | SLOC | missed |
   +------------------------------------------+----------+------+------+--------+
   | hello.js                                 |    87.50 |   10 |    8 |      1 |
   +------------------------------------------+----------+------+------+--------+
                                              |    87.50 |   10 |    8 |      1 |
                                              +----------+------+------+--------+

   hello.js:

      1 | 1 | var greeting = "Hello World!";
      2 | 1 | var farewell  = "Goodbye Cruel World!";
      3 | 1 | var greet = function() {
      4 | 1 |    return greeting;
      5 |   | };
      6 | 1 | var leave = function() {
      7 | 0 |    return farewell;
      8 |   | };
      9 | 1 | exports.greet = greet;
     10 | 1 | exports.leave = leave; 

Cool. It tells us we haven't tested the part about leaving with a farewell. All we have to do if we want 100% code coverage is to define a test for "leave":

// test/hello.test.js
var assert = require('assert'),
    hello = require('hello.js');

module.exports = {
   'greet()': function() {
       assert.equal('Hello World!', hello.greet());
   }, 
   'leave()': function() {
       assert.equal('Goodbye Cruel World!', hello.leave());
   }
};

And now when we run expresso again, we should have 100% code coverage.

$ expresso -I lib -c

  Test Coverage

   +------------------------------------------+----------+------+------+--------+
   | filename                                 | coverage | LOC  | SLOC | missed |
   +------------------------------------------+----------+------+------+--------+
   | hello.js                                 |   100.00 |   10 |    8 |      0 |
   +------------------------------------------+----------+------+------+--------+
                                              |   100.00 |   10 |    8 |      0 |
                                              +----------+------+------+--------+

   hello.js:

      1 | 1 | var greeting = "Hello World!";
      2 | 1 | var farewell  = "Goodbye Cruel World!";
      3 | 1 | var greet = function() {
      4 | 1 |    return greeting;
      5 |   | };
      6 | 1 | var leave = function() {
      7 | 1 |    return farewell;
      8 |   | };
      9 | 1 | exports.greet = greet;
     10 | 1 | exports.leave = leave; 

Cool, eh?

Expresso offers several other assert functions that you can find in the documentation here: http://visionmedia.github.com/expresso/.

A simple intro to NodeJS modules

In preparation for an upcoming workshop at Shad Valley on NodeJS, I tried to come up with some simple examples of the CommonJS module system in NodeJS.  Felix Geisendörfer has a similar "hello world" example in his Node.js Beginner Guide.  I've put the main concepts into a short tutorial. We start with one line of Javascript, and turn it into a working NodeJS module.

Assuming you have node installed, you can run "node" from the command line to get the REPL (ie. node shell).

$ node
> console.log("Hello World!");
Hello World!

Wonderful. How about putting this in a file?

// hello.js
console.log("Hello World!")

Then run:

$ node hello.js
Hello World!

Also wonderful. Then, how about making an application that includes hello.js? Let's call it app.js. Here are the two files:

// app.js
require('./hello.js');

// hello.js
console.log("Hello World!");

Now we can run it like this.

$ node app.js
Hello World!

Again wonderful. We've written a module. Not a very fancy one, but it's a start. Now, as an aside, suppose we want to run app.js like a shell script, without specifying "node" on the command line all the time? Simply add the node command to the app.js script:

#!/usr/bin/env node
// app.js
require('./hello.js');

Oh, and make it executable.

chmod u+x app.js

Now we can run it like so:

$ ./app.js
Hello World!

Now we can begin having fun. The thing about modules is that whatever you declare in the module (hello.js) is local to that module. If you want to use it externally, you have to add it to the special variable "exports":

// hello.js
var greeting = "Hello World!";
var greet = function() {
   console.log(greeting);
}
exports.greet = greet;

#!/usr/bin/env node
// app.js
var mygreet = require('./hello.js').greet;
mygreet();

This accomplishes the same thing, but take note of a couple things. First, "var greeting" is not available outside of the hello.js file. The function "greet" is available though, because we added it to exports. So that's how you get variables, functions and objects to be available from your module. Easy peasy.

jQuery on the Command Line

This is too cool... A Jquery command line interface. TJ Holowaychuk posted it to his git repo last night.

https://github.com/visionmedia/query

Example:

   $ curl http://twitter.com | query logging img attr alt
   Twitter