Mochitests is a test runner for integration (end-to-end) testing that allows us to test the debugger literally as a user would use it natively.

It is different from other test runners (Jest, Mocha, Jasmine) because it simulates a real user interacting with your application. For debugger, we do this by building your own version of Firefox with your changes to debugger. This Firefox build and its debugger is what's tested by Mochitests, in the same way that a user interacting with the debugger would natively.

If you've submitted a pull request to this project, you've already worked with Mochitests! TravisCI, which is one of the checks your PR has to pass, runs Mochitests every time you push a new commit to your PR.

• What does Mochitests do
• Initial Setup
• Running the tests
• Folder Overview
• Writing tests
• Your First Test
• Debugging your tests
• Test Writing Tips
• Adding New Test Files
• Debugging Intermittents
• Videos
• Appendix

Simply put, its tests simulate what you expect a user to do, and then check that the expected debugger behavior happens.

Typically, tests would simulate a user's action flow through the debugger (e.g. click this button, press this key). After performing a set of actions, we set assertions (e.g. debugger is paused/not paused, elements appear/disappear).

A typical test workflow would look like this.

Before starting, make sure you have the most updated versions of Homebrew and Python. Afterwards, run the following from inside the main debugger folder:

brew install mercurial
brew install autoconf@2.13 && brew unlink autoconf
./bin/prepare-mochitests-dev

On your first setup, ./bin/prepare-mochitests-dev clones Firefox's repository (mozilla-central). Downloading all of Firefox's source files may take ~30-60 minutes depending on your internet connection.

After your initial setup, running ./bin/prepare-mochitests-dev updates the cloned Firefox repository.

The detailed instructions for setting up your environment to build Firefox for Windows can be found here. Make sure to follow the steps under the sections Getting ready, Visual Studio 2017 and Required tools.

Afterwards, you can open a unix-flavor shell by starting:

C:\mozilla-build\start-shell.bat

In the shell, navigate to the debugger project folder. On your first setup, ./bin/prepare-mochitests-dev clones Firefox's repository (mozilla-central). Downloading all of Firefox's source files may take ~30-60minutes depending on your internet connection.

After your initial setup, running ./bin/prepare-mochitests-dev updates the cloned Firefox repository.

NOTE: Make sure you have Yarn installed before proceeding with the following steps

Every time you want to run your tests, the Firefox clone needs to be rebuilt with your updated local debugger. To do that, run the following:

yarn copy
./firefox/mach build

After rebuilding Firefox, you are ready to run your tests! There are three options:

// 1) Run the tests on your Firefox clone **without pausing** on debugger statements:
yarn mochi

// 2) Run the tests on your Firefox clone **with pausing** on debugger statements:
yarn mochid

// 3) Run the tests on your Firefox clone **headlessly without pausing** on debugger statements:
yarn mochih

The commands above will run all of your tests. If you want to only run one test, you can specify a test you want to run by adding its pattern. For example, if you want to run the test browser_dbg-tabs-keyboard.js headlessly, you would run yarn mochih tabs-keyboard.

A common testing workflow is 1) editing the test or the debugger code and 2) wanting to check whether a specific test passes. A common script to run for that scenario is yarn copy && ./firefox/mach build && yarn mochih your-test-name-here. This copies your debugger, rebuilds Firefox and runs your test all in one go.

NOTE: While running yarn mochi or yarn mochid, keep focus on the browser window while the tests are being run

All relevant Mochitest files are in tests/mochitests. A quick glance reveals the following:

• test files are all prefixed with browser_dbg-
• test runner helper functions are contained within helpers.js
• source files to be loaded by debugger test instances are in the folder examples/

Each individual test has the following structure in common - add_task() is provided with an async function in which the debugger is initialized:

add_task(async function() {
  const dbg = await initDebugger("page-to-be-loaded.html");

  /* Your test here */

});

Sometimes, your page-to-be-loaded.html may rely on external scripts. In that case, it's good practice to provide those sources to initDebugger() so that it knows to wait for those sources to be loaded before finishing the initialization.

add_task(async function() {
  const dbg = await initDebugger("page-to-be-loaded.html", "source.js", "another-source.js");

  /* Your test here */

});

NOTE: The root directory for any relative URL specified is test/mochitest/examples

At its core, what Mochitest does is it simulates a user action, like a mouse click on an element, and checks whether the expected behavior happens. Take for example the expand/collapse behavior of the Watch Expressions pane upon clicking its header:

1. By default, the Watch Expression pane is expanded
2. If we click on the Watch Expressions header, we expect the Watch Expression pane to collapse i.e. "content" is hidden
3. If we click on the Watch Expressions header again, we expect the Watch Expression pane to expand i.e. "content" is displayed

For illustration purposes, the structure of the Watch Expression pane element looks like this:

<li class="watch-expressions-pane">
  <h2 class = "_header">...</h2>
  <div class = "_content">...</div>
</li>

We might then write that test like this:

add_task(async function() {
  const dbg = await initDebugger("doc-watch-expressions.html");

  // 1) Check whether Watch Expressions pane is expanded
  var watchExpressionsContent = findElementWithSelector(dbg, ".watch-expressions-pane ._content");
  ok(watchExpressionsContent === true, "watch expressions content is displayed")

  // 2) Click on the header to collapse the pane. Check that it's collapsed
  const watchExpressionsHeader = findElementWithSelector(dbg, ".watch-expressions-pane ._header");
  watchExpressionsHeader.click();
  watchExpressionsContent = findElementWithSelector(dbg, ".watch-expressions-pane ._content");
  ok(watchExpressionsContent === false, "watch expressions content is not displayed")

  // 3) Click on the header again to expand the pane. Check that it's expanded
  watchExpressionsHeader.click();
  watchExpressionsContent = findElementWithSelector(dbg, ".watch-expressions-pane ._content");
  ok(watchExpressionsContent === true, "watch expressions content is displayed")
});

While this is a simple example, most of the Mochitests you will encounter follow the same process. What's important in writing them is to know clearly the actions you expect the user to do, how the debugger should behave in response to those actions, and insert assertions to test those behaviors.

In this example, we use a few helper functions that you may not be familiar with yet. These helper functions are all in test/mochitest/helpers.js, and for the most part are just variants on concepts you probably already know. For example, findElementWithSelector() is simply a wrapper for document.querySelector() specifically for your debugger instance.

While working on your tests, you might find that they're not running the way you expect them to. There are a few ways to figure out what exactly is happening and we will list them out here.

The first step for debugging an integration test is establishing a clear sequence of events: where did the test break and what were the events that happened before it broke. The Mochitests logs are a valuable source of information in providing this context. A quick glance at the logs may provide you with enough information to figure out what's gone wrong. These include:

• Actions that fired
• Assertions that passed/failed
• Events that the test waited for (e.g. dispatches, state changes)

We like to add additional logging in the test and debugger code with info() calls, which is similar to console.log(). Where necessary, use template literals with expressions to make info() calls as informative as possible.

• info("Adding an expression")
• info(`Thread event '${eventName}' fired.`)
• info(`Waiting for ${type} to dispatch ${eventRepeat} time(s)`)

It's worth noting that any console.log() statements in non-test code that runs in the debugger (e.g. React Component code) does not get outputted to the terminal. This is because those console.log() calls print their logs on the test browser console instead. If you need to print something to the terminal from non-test code, use dump() calls instead which functions similarly to console.log()

Sometimes you might want to stop the test execution to see what's going on.

In that case you can insert an await waitForever() within your test. This doesn't technically pause the debugger, but it stops the test execution all the same because the promise never gets resolved. This allows you to see what the DOM looks like at that point in time or inspect things via the debugger/console.

add_task(async function() {
  const dbg = await initDebugger("doc-watch-expressions.html");
  var watchExpressionsContent = findElementWithSelector(dbg, ".watch-expressions-pane ._content");

  // Stop the test here to inspect the DOM and see whether content really is displayed
  await waitForever();

  ok(watchExpressionsContent === true, "watch expressions content is displayed")
});

A similar alternative is to use await waitForTime(ms) which takes a millisecond(ms) value to wait for, instead of waiting indefinitely.

To pause the debugger, you need to put a debugger statement within your tests.

add_task(async function() {
  const dbg = await initDebugger("doc-watch-expressions.html");
  var watchExpressionsContent = findElementWithSelector(dbg, ".watch-expressions-pane ._content");

  // Pause the test here and use the debugger to look around
  debugger;

  ok(watchExpressionsContent === true, "watch expressions content is displayed")
});

This debugger statements is paused at only when you run the mochitests in debugger mode (yarn mochid your-test-here). While debugging with yarn mochid, a separate browser toolbox is opened from which you can debug the test that is currently running. While you can set breakpoints in this mode, they're not paused at - only inline debugger statements are paused at.

Note that these debugger statements are not limited to test code. They are also paused at in non-test code, which you can try by adding a debugger statement to a relevant React component of a test you will be running with yarn mochid.

By now, you should have a good grasp of the basic concepts of testing with Mochitests.

At this point, it's worth taking some time to become familiar with the helper methods in helpers.js. Below, we list some common patterns that come up while testing, and "best practices" for how to deal with them using the helper methods.

It's common to want to wait for something to happen in a test. Generally we wait for one of two things to happen:

1. Waiting for the Redux state to change

add_task(async function() {
  const dbg = await initDebugger("doc-sourcemaps.html", "entry.js");
  const entrySrc = findSource(dbg, "entry.js");

  await addBreakpoint(dbg, entrySrc, 5);
  await addBreakpoint(dbg, entrySrc, 13);
  await addBreakpoint(dbg, entrySrc, 15);

  // We use waitForState() here to make sure that our predicate function returns true before the is() assertion
  await waitForState(dbg, state => dbg.selectors.getBreakpointCount(state) === 3);
  is(getBreakpointCount(getState()), 3, "Three breakpoints exist");
});

2. Waiting for a Redux action to be dispatched

add_task(async function() {
  const dbg = await initDebugger("doc-scripts.html", "simple1");

  await selectSource(dbg, "simple1");
  is(countTabs(dbg), 1);
  pressKey(dbg, "close");

  // We use waitForDispatch() here to make sure that the CLOSE_TAB action has fired before our assertion counts the tabs
  // Otherwise our assertion may count the tabs before the tab is actually closed
  waitForDispatch(dbg, "CLOSE_TAB");
  is(countTabs(dbg), 0);
});

You can find elements defined in the debugger instance's DOM by using either:

1. findElement() which takes an element name from the shared selectors object defined in helpers.js.

add_task(async function() {
  const dbg = await initDebugger("doc-minified.html", "math.min.js");
  await selectSource(dbg, "math.min.js", 2);

  // We use findElement() here to find the pretty print button within the DOM.
  // The second argument passed (prettyPrintButton) should correspond to an Object key to the selector variable in helpers.js
  const prettyPrintButton = findElement(dbg, "prettyPrintButton");
  ok(Boolean(prettyPrintButton), "Pretty Print Button is hidden");
});

2. findElementWithSelector() which takes a selector.

add_task(async function() {
  const dbg = await initDebugger("doc-minified.html", "math.min.js");
  await selectSource(dbg, "math.min.js", 2);

  // We use findElement() here to find the pretty print button within the DOM.
  // The second argument passed (prettyPrintButton) should correspond to an Object key to the selector variable in helpers.js
  const prettyPrintButton = findElementWithSelector(dbg, ".source-footer .prettyPrint");
  ok(Boolean(prettyPrintButton), "Pretty Print Button is hidden");
});

If you want to evaluate a function in the debuggee context you can use the invokeInTab function. This invokes a function that is defined within the page itself.

invokeInTab(dbg, "doSomething");

You can invoke functions that are from a page's external script file, like you would in the console. For example, if the tab's page "myPage.html" loads an external script "script.js" and within "script.js" there is a function "myScriptFunction()", you can invoke it like this:

invokeInTab(dbg, "myScriptFunction")

You can also invoke document methods this way:

invokeInTab(dbg, "content.document.querySelector('.source-footer .prettyPrint')")

// You can also click a button this way
invokeInTab(dbg, "content.document.querySelector('.source-footer .prettyPrint').click()")

The dbg object returned by initDebugger() has several helpful properties, e.g., actions, selectors, getState, store, toolbox, win. Click here to learn more about these helpers.

If you add new tests files, make sure to list them in the browser.ini file. You will see the other test files there. Add a new entry with the same format as the others. You can also add new JS or HTML files by listing in under support-files.

Intermittents are when a test succeeds most the time (~95%) of the time, but not all the time. There are several easy traps that result in intermittents:

Sometimes the server is not as consistent as you would like. For example, reloading can sometimes cause sources to load out of order, or stepping too quickly can cause the debugger to enter a bad state.

A memorable example of this type of inconsistency came when debugging stepping behavior. It turns out that 1% of the time the browser toolbox will step into an unexpected location. The solution is to loosen our expectations :)

Sometimes action "B" can fire before action "A" is done. This is a race condition that can be hard to track down. When you suspect this might happen, it is a good practice to start listening for "B" before you fire action "A".

Here's an example where this happened with reloading.

One common way tests start failing occurs when the Redux actions introduces a new asynchronous operation. A good way to safe guard your tests is to wait on state to have certain values.

For example, a test that we fixed was pretty printing. The test initially waited for the "select source" action to fire, which wasn't predictable. Switching the test to wait for the formatted source to exist first simplified the test tremendously.

If you're looking for some tutorials on how to write and debug mochitests

• How We Test the Debugger
• Mochitest (Pause on Next)

The mochitest cli has a lot of advanced options that are worth learning about. Here is a quick intro in how it can be used

cd firefox
./mach mochitest devtools/client/debugger # runs all the debugger tests
./mach mochitest browser_dbg-editor-highlight # runs one test
./mach mochitest --jsdebugger browser_dbg-editor-highlight # runs one test with the browser toolbox open

Visit the mochitest MDN page to learn more about Mochitests and its more advanced arguments.

If symbolic link is suddenly lost between debugger and Firefox source, in your terminal, try the following

1. Navigate to the firefox directory (i.e. cd ~/debugger/firefox)
2. Execute ./mach mochitest --headless devtools/client/debugger

If a symbolic link occurs, error message(s) will be displayed.

3. Execute ./bin/prepare-mochitests-dev.

Running this command again allows the preparation script to check the integrity of the firefox directory and all symbolic links. It will then automatically execute yarn copy on your behalf, which ensures the symbolic linking process is complete.

4. On a new terminal tab, execute the command to run your test again. If this failed, proceed to step 5.

5. Execute ./mach configure

This will attempt to fix the harness' configurations.

If you are having issues running mochitest due to missing the Rust compiler, try the following:

1. In the root directory of the project (i.e. debugger/), execute ./mach configure.

2. Execute yarn watch

3. Open a new terminal tab and run your tests again. If this failed, proceed to step 4 and 5.

4. Execute ./mach bootstrap.

5. Execute ./bin/prepare-mochitests-dev.

You may see warnings along the way and the process may appear to be frozen. Please be patient, this is expected as it will take a while to recompile. Warning messages does not mean the compilation process has failed.

6. Repeat steps 2 and 3.