<table width="650">

<tr>

<td>

<p class="license">

This tutorial is from the book <a href="http://www.processing.org/learning/books/#shiffman">Learning Processing, 2nd Edition</a> by Daniel Shiffman, published by Morgan Kaufmann, &copy; 2015 Elsevier Inc. All rights reserved. If you see any errors or have comments, please <a href="https://github.com/processing/processing-docs/issues?state=open">let us know</a>.

</p>

<h1 style="line-height: 0.7em;">Data</h1>

<h3 style="line-height: 0.7em;"><em>Daniel Shiffman</em></h3>

<p>

This tutorial picks up where the <a href="http://processing.org/learning/text/">Strings and Drawing Text</a> tutorial leaves off and examines how to use String objects as the basis for reading and writing data. We'll start by learning more sophisticated methods for manipulating Strings, searching in them, chopping them up, and joining them together. Afterwards, we'll see how these skills allow us to use input from data sources, such as text files, web pages, xml feeds, and 3rd party APIs and take a step into the world of data visualization.

</p>

<h3>Manipulating Strings</h3>

<p>

In <a href="http://processing.org/learning/text/">Strings and Drawing Text</a>, we touched on a few of the basic functions available in the Java <tt>String</tt>, such as <tt>charAt()</tt>, <tt>toUpperCase()</tt>, <tt>equals()</tt>, and <tt>length()</tt>. These functions are documented on the Processing reference page for Strings. Nevertheless, in order to perform some more advanced data parsing techniques, we'll need to explore some additional String manipulation functions <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html">documented in the Java API</a>.

<br /><br />

Let's take a closer look at the following two String functions: <tt>indexOf()</tt> and <tt>substring()</tt>.<br /><br />

<tt>indexOf()</tt> locates a sequence of characters within a string. It takes one argument — a search string — and returns a numeric value that corresponds to the first occurrence of the search string inside of the <tt>String</tt> object being searched.<br />

String search = "def";

String toBeSearched = "abcdefghi";

int index = toBeSearched.indexOf(search); // The value of index in this example is 3.

Strings are just like arrays, in that the first character is index number zero and the last character is the length of the string minus one. If the search string cannot be found, <tt>indexOf()</tt> returns <tt>-1</tt>. This is a good choice because <tt>-1</tt> is not a legitimate index value in the string itself, and therefore can indicate "not found." There are no <em>negative</em> indices in a string of characters or in an array.

<br /><br />

After finding a search phrase within a string, we might want to separate out part of the string, saving it in a different variable. A part of a string is known as a <em>substring</em> and substrings are made with the <tt>substring()</tt> function which takes two arguments, a start index and an end index. <tt>substring()</tt> returns the substring in between the two indices.<br />

String alphabet = "abcdefghi";

String sub = alphabet.substring(3, 6); // The String sub is now "def".

Note that the substring begins at the specified <em>start index</em> (the first argument) and extends to the character at <em>end index</em> (the second argument) <em>minus one</em>. I know, I know. Wouldn’t it have been easier to just take the <em>substring</em> from the start index all the way to the end index? While this might initially seem true, it’s actually quite convenient to stop at end index minus one. For example, if you ever want to make a substring that extends to the end of a string, you can simply go all the way to <tt>thestring.length()</tt>. In addition, with end index minus one marking the end, the length of the substring is easily calculated as <em>end index minus begin index</em>.

</p>

<h3>Splitting and Joining Strings</h3>

<p>

In <a href="http://processing.org/learning/text/">Strings and Drawing Text</a>, we saw how strings can be joined together (referred to as "concatenation") using the "+" operator. Let's review with a example that uses concatenation to get user input from a keyboard.

<br /><br />

<img src="imgs/fig_18_01_user_input.png" class="tut" />

PFont f;

// Variable to store text currently being typed

String typing = "";

// Variable to store saved text when return is hit

String saved = "";

void setup() {

size(300, 200);

f = createFont("Arial", 16);

}

void draw() {

background(255);

int indent = 25;

// Set the font and fill for text

textFont(f);

fill(0);

// Display everything

text("Click in this sketch and type. \nHit return to save what you typed.", indent, 40);

text(typing, indent, 90);

text(saved, indent, 130);

}

void keyPressed() {

// If the return key is pressed, save the String and clear it

if (key == '\n') {

saved = typing;

typing = "";

// Otherwise, concatenate the String

} else {

typing = typing + key;

}

}

</pre>

Processing has two additional functions that make joining strings (or the reverse, splitting them up) easy. In sketches that involve parsing data from a file or the web, you might get hold of that data in the form of an array of strings or as one long string. Depending on what you want to accomplish, it’s useful to know how to switch between these two modes of storage. This is where these two new functions, <tt>split()</tt> and <tt>join()</tt>, will come in handy.

</p>

<strong><em>"one long string or array of strings" ←→ {"one", "long", "string", "or" ,"array", "of", "strings"}</em></strong>

<p>

Let’s take a look at the <tt>split()</tt> function. <tt>split()</tt> separates a longer string into an array of strings, based on a split character known as the <em>delimiter</em>. It takes two arguments, the <tt>String</tt> object to be split and the delimiter. (The delimiter can be a single character or a string.) In the code below, the period is not set as a delimiter and therefore will be included in the last string in the array: “dog.” Note how <tt>printArray()</tt> can be used to print the contents of an array and their corresponding indices to the message console. <br />

// Splitting a string based on spaces

String spaceswords = "The quick brown fox jumps over the lazy dog.";

String[] list = split(spaceswords, " ");

printArray(list);

Here is an example using a comma as the delimiter (this time passing in a single character: <tt>','</tt>.)

<br />

// Splitting a string based on commas

String commaswords = "The,quick,brown,fox,jumps,over,the,lazy,dog.";

String[] list = split(commaswords, ",");

printArray(list);

If you want to use more than one delimiter to split up a text, you must use the Processing function <tt>splitTokens()</tt>. <tt>splitTokens()</tt> works identically as <tt>split()</tt> with one exception: any character that appears in the passed string qualifies as a delimiter.) In the code below, the period is specified as a delimiter and therefore will not be included in the last string in the array: “dog”.<br />

// Splitting a String based on multiple delimiters

String stuff = "hats & apples, cars + phones % elephants dog.";

String[] list = splitTokens(stuff, " &,+." );

printArray(list);

If you are splitting numbers in a string, the resulting array can be converted into an integer array with Processing’s <tt>int()</tt> function. Numbers in a string are not numbers and cannot be used in mathematical operations unless you convert them first.<br />

// Calculate sum of a list of numbers in a String

String numbers = "8,67,5,309";

// Converting the String array to an int array

int[] list = int(split(numbers, ','));

int sum = 0;

for (int i = 0; i&lt;list.length; i++ ) {

sum = sum + list[i];

}

println(sum);

The reverse of <tt>split()</tt> is <tt>join()</tt>. <tt>join()</tt> takes an array of strings and joins them together into one long <tt>String</tt> object. The <tt>join()</tt> function also takes two arguments, the array to be joined and a <em>separator</em>. The separator can either be a single character or a string of characters.

</p>

<p>

Consider the following array:<br />

String[] lines = {"It", "was", "a", "dark", "and", "stormy", "night."};

Using the “+” operator along with a <code>for</code> loop, you can join a string together as follows:<br />

// Manual Concatenation

String onelongstring = "";

for (int i = 0; i &lt; lines.length; i++) {

onelongstring = onelongstring + lines[i] + " ";

}

The <tt>join()</tt> function, however, allows you to bypass this process, achieving the same result in only one line of code.<br />

// Using Processing's join()

String onelongstring = join(lines, " ");

<h3>Dealing with Data</h3>

<p>

Data can come from many different places: websites, news feeds, spreadsheets, databases, and so on. Let's say you've decided to make a map of the world's flowers. After searching online you might find a PDF version of a flower encyclopedia, or a spreadsheet of flower genera, or a JSON feed of flower data, or a REST API that provides geolocated lat/lon coordinates, or some web page someone put together with beautiful flower photos, and so on and so forth. The question inevitably arises: “I found all this data; which should I use, and how do I get it into Processing?”

</p>

<p>

If you are really lucky, you might find a <a href="http://processing.org/reference/libraries/">Processing library</a> that hands data to you directly with code. Maybe the answer is to just download this library and write some code like:<br />

import flowers.*;

void setup() {

FlowerDatabase fdb = new FlowerDatabase();

Flower sunflower = fdb.findFlower("sunflower");

float h = sunflower.getAverageHeight();

}

In this case, someone else has done all the work for you. They've gathered data about flowers and built a Processing library with a set of functions that hands you the data in an easy-to-understand format. This library, sadly, does not exist (not yet), but there are some that do. For example, <a href="http://www.onformative.com/lab/google-weather-library-for-processing">YahooWeather</a> is a library by Marcel Schwittlick that grabs weather data from Yahoo for you, allowing you to write code like <tt>weather.getWindSpeed()</tt> or <tt>weather.getSunrise()</tt> and more. There is still plenty of work to do in the case of using a library.

</p>

<p>

Let's take another scenario. Say you’re looking to build a visualization of Major League Baseball statistics. You can't find a Processing library to give you the data but you do see everything you’re looking for at mlb.com. If the data is online and your web browser can show it, shouldn't you be able to get the data in Processing? Passing data from one application (like a web application) to another (say, your Processing sketch) is something that comes up again and again in software engineering. A means for doing this is an API or “application programming interface”: a means by which two computer programs can talk to each other. Now that you know this, you might decide to search online for “MLB API”. Unfortunately, mlb.com does not provide its data via an API. In this case you would have to load the raw source of the website itself and manually search for the data you’re looking for. While possible, this solution is much less desirable given the considerable time required to read through the HTML source as well as program algorithms for parsing it.

</p>

<p>

Each means of getting data comes with its own set of challenges. The ease of using a Processing library is dependent on the existence of clear documentation and examples. But in just about all cases, if you can find your data in a format designed for a computer (spreadsheets, XML, JSON, etc.), you'll be able to save some time in the day for a nice walk outside.

</p>

<p>

One other note worth a mention about working with data. When developing an application that involves a data source, such as a data visualization, it’s sometimes useful to develop with “dummy” or “fake” data. You don't want to be debugging your data retrieval process at the same time as solving problems related to algorithms for drawing. In keeping with my <em>one-step-at-a-time</em> mantra, once the meat of the program is completed with dummy data, you can then focus solely on how to retrieve the actual data from the real source. You can always use random or hard-coded numbers into your code when you’re experimenting with a visual idea and connect the real data later.</p>

<h3>Working with Text Files</h3>

<p>

Let's begin by working with the simplest means of data retrieval: reading from a text file. Text files can be used as a very simple database (you could store settings for a program, a list of high scores, numbers for a graph, etc.) or to simulate a more complex data source.

</p>

<p>

In order to create a text file, you can use any simple text editor. Windows Notepad or Mac OS X TextEdit will do; just make sure you format the file as “plain text.” It is also advisable to name the text files with the “.txt” extension, to avoid any confusion. And just as with image files, these text files should be placed in the sketch’s “data” directory in order for them to be recognized by the Processing sketch.</p>

<p>

Once the text file is in place, Processing’s <tt>loadStrings()</tt> function is used to read the content of the file into a <tt>String</tt> array. The individual lines of text in the file each become an individual element in the array.

<br /><br />

<img src="imgs/fig_18_02_filetxt.png" class="tut" />

// This code will print all the lines from the source text file.

String[] lines = loadStrings("file.txt");

println("There are " + lines.length + " lines.");

printArray(lines);

To run the code, create a text file called “file.txt,” type a bunch of lines in that file, and place it in your sketch’s data directory.<br /><br />Text from a file can be used to generate a simple visualization. Take the following data file.

<br /><br />

<img src="imgs/fig_18_03_datatxt.png" class="tut" /><br /><br />

The results of visualizing this data are shown below.</p>

<strong>Graphing Comma-Separated Numbers from a Text File</strong>

<br /><br />

<img src="imgs/fig_18_04_bargraph.png" class="tut" />

int[] data;

void setup() {

size(200, 200);

// Load text file as a String

String[] stuff = loadStrings("data.csv");

// Convert string into an array of integers using ',' as a delimiter

data = int(split(stuff[0], ','));

}

void draw() {

background(255);

stroke(0);

for (int i = 0; i&lt;data.length; i++) {

// Use array of ints to set the color and height of each rectangle.

rect(i*20, 0, 20, data[i]);

}

noLoop();

}

Looking at how to parse a csv file with <tt>split()</tt> was a nice learning exercise. In truth, dealing with csv files (which can easily be generated from spreadsheet software such as Google docs) is such a common activity that Processing has an entire built-in class called <tt>Table</tt> to handle the parsing for you.

</p>

<h3>Tabular Data</h3>

<p>

A table consists of data arranged as a set of rows and columns, also called “tabular data.” If you've ever used a spreadsheet, this is tabular data. Processing's <tt>loadTable()</tt> function takes comma-separated (csv) or tab-separated (tsv) values and automatically places the contents into a <tt>Table</tt> object storing the data in columns and rows. This is a great deal more convenient than struggling to manually parse large data files with <tt>split()</tt>. It works as follows. Let's say you have a data file that looks like:

</p>

<br />

<img src="imgs/fig_18_05_datacsv.png" class="tut" />

<p>

Instead of saying:<br />

String[] stuff = loadStrings("data.csv");

We can now say:<br />

Table table = loadTable("data.csv");

Now I've missed an important detail. Take a look again at the data.csv text file above. Notice how the first line of text is not the data itself, but rather a <em>header row</em>. This row includes labels that describe the data included in each subsequent row. The good news is that Processing can automatically interpret and store the headers for you, if you pass in the option <tt>"header"</tt> when loading the table. (In addition to <tt>"header"</tt>, there are other options you can specify. For example, if your file is called data.txt but is comma separated data you can pass in the option <tt>"csv"</tt>. If it also has a header row, then you can specifiy both options like so: <tt>"header,csv"</tt>). A full list of options can be found on the <a href="http://processing.org/reference/loadTable_.html"><tt>loadTable()</tt> documentation page</a>.<br />

Table table = loadTable("data.csv", "header");

Now that the table is loaded, I can show how you grab individual pieces of data or iterate over the entire table. Let's look at the data visualized as a grid.

<br />

<img src="imgs/data_05_headers.jpg" class="tut" />

<br />

In the above grid you can see that the data is organized in terms of rows and columns. One way to access the data, therefore, would be to request a value by its numeric row and column location (with zero being the first row or first column). This is similar to accessing a pixel color at a given (x,y) location, though in this case the y position (row) comes first. The following code requests a piece of data at a given (row, column) location.<br />

int val1 = table.getInt(2, 1); // val now has the value 235

float val2 = table.getFloat(3, 2); // val2 now has the value 44.758068

String s = table.getString(0, 3); // s now has the value “Happy”

While the numeric index is sometimes useful, it’s generally going to be more convenient to access each piece of data by the column name. For example, I could pull out a specific row from the <tt>Table</tt>.<br />

TableRow row = table.getRow(2); // Gets the third row (index 2)

Note in the above line of code that a <tt>Table</tt> object refers to the entire table of data while a <a href="http://processing.org/reference/TableRow.html"><tt>TableRow</tt></a> object handles an individual row of data within the <tt>Table</tt>.

</p>

<p>

Once I have the <tt>TableRow</tt> object, I can ask for data from some or all of the columns.<br />

int x = row.getInt("x"); // // x has the value 273

int y = row.getInt("y"); // y has the value 235

float d = row.getFloat("diameter"); // d has the value 61.14072

String s = row.getString("name"); // s has the value “Joyous”

The method <a href="http://processing.org/reference/Table_getRow_.html"><tt>getRow()</tt></a> returns a single row from the table. If you want to grab all the rows and iterate over them you can do so in a loop with a counter accessing each row one at a time. The total number of available rows can be retrieved with <tt>getRowCount()</tt>.<br />

for (int i = 0; i&lt;table.getRowCount(); i++) {

// Access each row of the table one at a time, in a loop.

TableRow row = table.getRow(i);

float x = row.getFloat("x");

float y = row.getFloat("y");

float d = row.getFloat("diameter");

String n = row.getString("name");

// Do something with the data of each row

}

If you want to search for a select number of rows within the table, you can do so with <a href="http://processing.org/reference/Table_findRows_.html"><tt>findRows()</tt></a> and <a href="http://processing.org/reference/Table_matchRows_.html"><tt>matchRows()</tt></a>.

</p>

<p>

In addition to being read, <tt>Table</tt> objects can be altered or created on the fly while a sketch is running. Cell values can be adjusted, rows can be removed, and new rows can be added. For example, to set new values in a cell there are functions <tt>setInt()</tt>, <tt>setFloat()</tt>, and <tt>setString()</tt>.<br />

row.setInt("x", mouseX); // Update the value of column "x" to mouseX in a given TableRow.

To add a new row to a <tt>Table</tt>, simply call the method <a href="http://processing.org/reference/Table_addRow_.html"><tt>addRow()</tt></a> and set the values of each column.<br />

//Create a new row.

TableRow row = table.addRow();

//Set the values of all columns in that row.

row.setFloat("x", mouseX);

row.setFloat("y", mouseY);

row.setFloat("diameter", random(40, 80));

row.setString("name", "new label");

To delete a row, simply call the method <a href="http://processing.org/reference/Table_removeRow_.html"><tt>removeRow()</tt></a> and pass in the numeric index of the row you would like removed. For example, the following code removes the first row whenever the size of the table is greater than ten rows.<br />

// If the table has more than 10 rows

if (table.getRowCount()&gt;10) {

//Delete the first row (index 0).

table.removeRow(0);

}

The following example puts all of the above code together. Notice how each row of the table contains the data for a <tt>Bubble</tt> object.

</p>

<strong>Loading and Saving Tabular Data</strong>

<br />

<img src="imgs/fig_18_07_tablebubbles.png" class="tut" />

// The data from the Table object will fill the array of Bubble objects

Table table;

Bubble[] bubbles;

void setup() {

size(480, 360);

loadData();

}

void draw() {

background(255);

// Display all bubbles

for (int i = 0; i&lt;bubbles.length; i++) {

bubbles[i].display();

}

}

void loadData() {

// "header" indicates the file has header row. The size of the array

// is then determined by the number of rows in the table.

table = loadTable("data.csv", "header");

bubbles = new Bubble[table.getRowCount()];

for (int i = 0; i&lt;table.getRowCount(); i++) {

// Iterate over all the rows in a table.

TableRow row = table.getRow(i);

// Access the fields via their column name (or index).

float x = row.getFloat("x");

float y = row.getFloat("y");

float d = row.getFloat("diameter");

String n = row.getString("name");

// Make a Bubble object out of the data from each row.

bubbles[i] = new Bubble(x, y, d, n);

}

}

void mousePressed() {

// When the mouse is pressed, create a new row and set the values for each column of that row.

TableRow row = table.addRow();

row.setFloat("x", mouseX);

row.setFloat("y", mouseY);

row.setFloat("diameter", random(40, 80));

row.setString("name", "Blah");

// If the table has more than 10 rows, delete the oldest row.

if (table.getRowCount()&gt;10) {

table.removeRow(0);

}

// This writes the table back to the original CSV file

// and reloads the file so that what's drawn matches.

saveTable(table, "data/data.csv");

loadData();

}

// This simple Bubble class draws a circle to the window

// and displays a text label when the mouse hovers.

class Bubble {

float x, y;

float diameter;

String name;

boolean over = false;

// Create the Bubble

Bubble(float tempX, float tempY, float tempD, String s) {

x = tempX;

y = tempY;

diameter = tempD;

name = s;

}

// Checking if mouse is over the bubble

void rollover(float px, float py) {

float d = dist(px, py, x, y);

if (d&lt;diameter/2) {

over = true;

} else {

over = false;

}

}

// Display the Bubble

void display() {

stroke(0);

strokeWeight(2);

noFill();

ellipse(x, y, diameter, diameter);

if (over) {

fill(0);

textAlign(CENTER);

text(name, x, y+diameter/2+20);

}

}

}

<p>

Here, the distance between a given point and a circle's center is compared to that circle's radius as depicted:

</p>

<br />

<img src="imgs/fig_18_08_rollover.png" class="tut" />

<p>

In the code below, the function returns a boolean value (true or false) depending on whether the point (mx,my) is inside the circle. Notice how radius is equal to half the diameter.

</p>

boolean rollover(int mx, int my) {

if (dist(mx, my, x, y)&lt;diameter/2) {

return true;

} else {

return false;

}

}

<h3>Data that is not in a Standardized Format</h3>

<p>

What if your data is not in a standard format like a table, how do you deal with it then? One of the nice features about <tt>loadStrings()</tt> is that in addition to pulling text from a file, you can also grab a URL. For example:<br />

String[] lines = loadStrings("http://www.yahoo.com");

When you send a URL path into <tt>loadStrings()</tt>, you get back the raw HTML (Hypertext Markup Language) source of the requested web page. It’s the same stuff that appears upon selecting “View Source” from a browser’s menu options. You don’t need to be an HTML expert to follow this section, but if you are not familiar at all with HTML, you might want to read <a href="http://en.wikipedia.org/wiki/HTML">http://en.wikipedia.org/wiki/HTML</a>.

</p>

<p>

Unlike with the comma-delimited data from a text file that was specially formatted for use in a Processing sketch, it’s not practical to have the resulting raw HTML stored in an array of strings (each element representing one line from the source). Converting the array into one long string can make things a bit simpler. As you saw earlier in the chapter, this can be achieved using <tt>join()</tt>.<br />

String onelongstring = join(lines, " ");

When pulling raw HTML from a web page, it’s likely you do not want all of the source, but just a small piece of it. Perhaps you’re looking for weather information, a stock quote, or a news headline. You can take advantage of the text manipulation functions you learned — <tt>indexOf()</tt>, <tt>substring()</tt>, and <tt>length()</tt> — to find pieces of data within a large block of text. Take, for example, the following <tt>String</tt> object:<br />

String stuff = "Number of apples:62. Boy, do I like apples or what!";

Let’s say I want to pull out the number of apples from the above text. My algorithm would be as follows:

<ol>

<li>

<p>Find the <em>end of the substring</em> “apples:” Call it start.</p>

</li>

<li>

<p>Find the <em>first period</em> after “apples:” Call it end.</p>

</li>

<li>

<p>Make a <em>substring</em> of the characters between start and end.</p>

</li>

<li>

<p>Convert the string to a number (if I want to use it as such).</p>

</li>

<p>

In code, this looks like:<br />

int start = stuff.indexOf("apples:" ) + 7; // STEP 1

// The index where a string ends can be found by

// searching for that string and adding its length (here, 8).

int end = stuff.indexOf(".", start); // STEP 2

String apples = stuff.substring(start, end); // STEP 3

int apple_no = int(apples); // STEP 4

The above code will do the trick, but I should be a bit more careful to make sure I don’t run into any errors if I do not find the string I am searching for. I can add some error checking and generalize the code into a function:<br />

// A function that returns a substring between two substrings.

// If the beginning of end "tag" is not found, the function returns an empty string.

String giveMeTextBetween(String s, String startTag, String endTag) {

// Find the index of the beginning tag

int startIndex = s.indexOf(startTag);

// If I don't find anything

if (startIndex == -1) {

return "";

}

// Move to the end of the beginning tag

startIndex += startTag.length();

// Find the index of the end tag

int endIndex = s.indexOf(endTag, startIndex);

// If I don't find the end tag,

if (endIndex == -1) {

return "";

}

// Return the text in between

return s.substring(startIndex, endIndex);

}

With this technique, you are ready to connect to a website from within Processing and grab data to use in your sketches. For example, you could read the HTML source from nytimes.com and look for today’s headlines, search finance.yahoo.com for stock quotes, count how many times the word “flower” appears on your favorite blog, and so on. However, HTML is an ugly, scary place with inconsistently formatted pages that are difficult to reverse engineer and parse effectively. Not to mention the fact that companies change the source code of web pages rather often, so any example that I might make while I am writing this paragraph might break by the time you read this paragraph.

</p>

<p>

For grabbing data from the web, an XML (Extensible Markup Language) or JSON (JavaScript Object Notation) feed will prove to be more reliable and easier to parse. Unlike HTML (which is designed to make content viewable by a human’s eyes) XML and JSON are designed to make content viewable by a computer and facilitate the sharing of data across different systems. Most data (news, weather, and more) is available this way, and I will look at examples in #beginner_xml and #JSON. Though much less desirable, manual HTML parsing is still useful for a couple reasons. First, it never hurts to practice text manipulation techniques that reinforce key programming concepts. But more importantly, sometimes there is data you really want that is not available in an API format, and the only way to get it is with such a technique. (I should also mention that regular expressions, an incredibly powerful techinque in text pattern matching, could also be employed here. As much as I love regex, it’s unfortunately beyond the scope of this tutorial.)

</p>

<p>

An example of data only available as HTML is the <a href="http://imdb.com">Internet Movie Database</a>. IMDb contains information about movies sorted by year, genre, ratings, etc. For each movie, you can find the cast and crew list, a plot summary, running time, a movie poster image, the list goes on. However, IMDb has no API and does not provide its data as XML or JSON. Pulling the data into Processing therefore requires a bit of detective work. Let's look at the page for the <em>Shaun the Sheep Movie</em>

</p>

<img src="imgs/fig_18_09_shaunsheep.png" class="tut" />

<p>

Looking in the HTML source from the above URL, I find a giant mess of markup.

</p>

<img src="imgs/fig_18_10_shaunsheep_sourc.png" class="tut" />

<p>

It’s up to me to pore through the raw source and find the data I am looking for. Let's say I want to know the running time of the movie and grab the movie poster image. After some digging, I find that the movie is 139 minutes long as listed in the following HTML.<br />

&lt;div class="txt-block"&gt;

&lt;h4 class="inline"&gt;Runtime:&lt;/h4&gt;

&lt;time itemprop="duration" datetime="PT139M"&gt;139 min&lt;/time&gt;

&lt;/div&gt;

For any given movie, the running time itself will be variable, but the HTML structure of the page will stay the same. I can therefore deduce that running time will always appear in between:<br />

&lt;time itemprop="duration" datetime="PT139M"&gt;

and:<br />

&lt;/time&gt;

Knowing where the data starts and ends, I can use <tt>giveMeTextBetween()</tt> to pull out the running time. A quote in Java marks the beginning or end of a string. So how do you include an actual quote in a String object? The answer is via an “escape” sequence. A quote can be included using a backward slash, followed by a quote. For example: String q = "This String has a quote \"in it";<br />

String url = "http://www.imdb.com/title/tt0058331";

String[] lines = loadStrings(url);

// Get rid of the array in order to search the whole page

String html = join(lines, " ");

// Searching for running time

String start = "<time itemprop=\"duration\" datetime=\"PT139M\">";

String end = "</time>";

String runningtime = giveMeTextBetween(html, start, end);

println(runningtime);

The following code retrieves both the running time and movie poster iamge from IMDb and displays it onscreen.

</p>

<strong>Parsing IMDb Manually</strong>

<br /><br />

<img src="imgs/fig_18_11_parsing_imdb.png" class="tut" />

String runningtime;

PImage poster;

void setup() {

size(300, 350);

loadData();

}

void draw() {

// Display all the stuff I want to display

background(255);

image(poster, 10, 10, 164, 250);

fill(0);

text("Shaun the Sheep", 10, 300);

text(runningtime, 10, 320);

}

void loadData() {

String url = "http://www.imdb.com/title/tt2872750/";

// Get the raw HTML source into an array of strings (each line is one element in the array).

// The next step is to turn array into one long string with join().

String[] lines = loadStrings(url);

String html = join(lines, "");

String start = "<time itemprop=\"duration\" datetime=\"PT139M\">";

String end = "</time>";

runningtime = giveMeTextBetween(html, start, end);Searching for running time.

start = "<link rel='image_src' href=\"";

end = "\">";

// Search for the URL of the poster image.

String imgUrl = giveMeTextBetween(html, start, end);

// Now, load that image!

poster = loadImage(imgUrl);

}

String giveMeTextBetween(String s, String before, String after) {

// This function returns a substring between two substrings (before and after).

// If it can’t find anything it returns an empty string.

String found = "";

// Find the index of before

int start = s.indexOf(before);

if (start == -1) {

return "";

}

// Move to the end of the beginning tag

// and find the index of the "after" String

start += before.length();

int end = s.indexOf(after, start);

if (end == -1) {

return "";

}

// Return the text in between

return s.substring(start, end);

}

<h3>Text Analysis</h3>

<p>

Loading text from a URL need not only be an exercise in parsing out small bits of information. It’s possible with Processing to analyze large amounts of text found on the web from news feeds, articles, and speeches, to entire books. A nice source is <a href="http://www.gutenberg.org/"><em>Project Gutenberg</em></a> which makes available thousands of public domain texts. Algorithms for analyzing text merits an entire book itself, but let’s look at some basic techniques.

</p>

<p>

A text concordance is an alphabetical list of words that appear in a book or body of text along with contextual information. A sophisticated concordance might keep a list of where each word appears (like an index) as well as which words appear next to which other words. In this case, I'm going to create a simple concordance, one that simply stores a list of words and their corresponding counts, i.e., how many times they appeared in the text. Concordances can be used for text analysis applications such as spam filtering or sentiment analysis. To accomplish this task, I am going to use the Processing built-in class <tt>IntDict</tt>.

</p>

<p>

As you learned earlier, an array is an ordered list of variables. Each element of the array is numbered and be accessed by its numeric index.

</p>

<img src="imgs/fig_18_12_traditional_arra.png" class="tut" />

<p>

However, what if instead of numbering the elements of an array you could name them? This element is named “Sue,” this one “Bob,” this one “Jane,” and so on and so forth. In programming, this kind of data structure is often referred to as an <em>associative array, map,</em> or <em>dictionary.</em> It’s a collection of (key, value) pairs. Imagine you had a dictionary of people's ages. When you look up “Sue” (the key), the definition, or value, is her age, 24.

</p>

<img src="imgs/fig_18_13_associative_arra.png" class="tut" />

<p>

Associative arrays can be incredibly convenient for various applications. For example, you could keep a list of student IDs (student name, id) or a list of prices (product name, price) in a dictionary. Here a dictionary is the perfect data structure to hold the concordance. Each element of the dictionary is a word paired with its count.

</p>

<p>

While there are many classes in Java for dealing with advanced data structures like maps, Processing provides you with a set of three built-in dictionary classes that are easy to use: <tt>IntDict</tt>, <tt>FloatDict</tt>, and <tt>StringDict</tt>. In all of these classes, the key is always a string while the value is variable (an integer, floating point number, or String). For the concordance, I'll use an <tt>IntDict</tt>.

</p>

<p>

Creating an <tt>IntDict</tt> is as easy as calling an empty constructor. Let's say you want a dictionary to keep track of an inventory of supplies.<br />

IntDict inventory = new IntDict();

Values can be paired with their keys using the <tt>set()</tt> method.<br />

// set() assigns an integer to a String.

inventory.set("pencils", 10);

inventory.set("paper clips", 128);

inventory.set("pens, 16");

There are a variety of other methods that can be called to change the value associated with a particular key. For example, if you wanted to add five pencils, you can use <tt>add()</tt>.<br />

inventory.add("pencils", 5); // The value of “pencils” is now 15.

A particularly convenient method for the concordance example is <tt>increment()</tt> which adds one to a key's value.<br />

inventory.increment("pens"); // The value of “pencils” is now 16.

To retrieve a value associated with a particular key, the <tt>get()</tt> method is used.<br />

int num = inventory.get("pencils"); // The value of num is 16.

Finally, dictionaries can be sorted by their keys (alphabetical) or values (smallest to largest or the reverse) with the methods <tt>sortKeys()</tt>, <tt>sortKeysReverse()</tt>, <tt>sortValues()</tt>, and <tt>sortValuesReverse()</tt>.

</p>

<p>

The concordance now becomes a rather simple program to write. All I need to do is load in a text file, chop it into words with <tt>splitTokens()</tt> and call <tt>increment()</tt> on an <tt>IntDict</tt> for every single word found in the text. The following example does precisely this with the entire text of Shakespeare’s play, <em>A Midsummer Night's Dream</em>, displaying a simple graph of the most used words.

</p>

<strong>Text Concordance Using IntDict</strong>

<br /><br />

<img src="imgs/fig_18_14_concordance_viz.png" class="tut" />

String[] allwords;

// Any punctuation is used as a delimiter.

String delimiters = " ,.?!;:[]";

IntDict concordance;

void setup() {

size(360, 640);

// Load A Midsummer Night's Dream into an array of strings

String url = "http://www.gutenberg.org/cache/epub/1514/pg1514.txt";

String[] rawtext = loadStrings(url);

// Join the big array together as one long string

String everything = join(rawtext, "" );

// All the lines in A Midsummer Night's Dream are first joined as one big string

// and then split up into an array of individual words.

// Note the use of splitTokens() since I am using spaces and punctuation marks as delimiters.

allwords = splitTokens(everything, delimiters);

// Make a new empty dictionary

concordance = new IntDict();

for (int i = 0; i&lt;allwords.length; i++) {

// It's useful to convert each word to lower case so that,

// for example, “The” and “the” are both counted as the same word.

String s = allwords[i].toLowerCase();

// For every single word, increase its count in the dictionary.

concordance.increment(s);

}

// Sort the dictionary so that words that appear most often are first.

concordance.sortValuesReverse();

}

void draw() {

background(255);

// Display the text and total times the word appears

int h = 20;

// In order to iterate over every word in the dictionary,

// first ask for an array of all of the keys.

String[] keys = concordance.keyArray();

for (int i = 0; i&lt;height/h; i++) {

// Look at each key one at a time and retrieve its count.

String word = keys[i];

int count = concordance.get(word);

fill(51);

// Displaying a rectangle along with the count as a simple graph.

rect(0, i*20, count/4, h-4);

fill(0);

text(word + ": " + count, 10+count/4, i*h+h/2);

stroke(0);

}

}

<p>

Processing also includes three classes for lists of numbers and strings: <tt>IntList</tt>, <tt>FloatList</tt>, and <tt>StringList</tt>. In other words, if you just want a list of words (without their counts) you could use a <tt>StringList</tt> rather than an <tt>IntDict.</tt>

</p>

<p>

If your data is available via a standardized format such as XML or JSON, the process of manually searching through text for individual pieces of data is no longer required. XML is designed to facilitate the sharing of data across different systems, and you can retrieve that data using the built-in Processing XML class.

</p>

<p>

XML organizes information in a tree structure. Let’s imagine a list of students. Each student has an ID number, name, address, email, and telephone number. Each student’s address has a city, state, and zip code. An XML tree for this dataset might look like the following:

</p>

<img src="imgs/data_06_xml.jpg" class="tut" />

<p>

The XMl source itself (with two students listed) is:<br />

&lt;?xml version = "1.0" encoding = "UTF-8 "?&gt;

&lt;students&gt;

&lt;student&gt;

&lt;id&gt;001&lt;/id&gt;

&lt;name&gt;Daniel Shiffman&lt;/name&gt;

&lt;phone&gt;555-555-5555&lt;/phone&gt;

&lt;email&gt;daniel@shiffman.net&lt;/email&gt;

&lt;address&gt;

&lt;street&gt;123 Processing Way&lt;/street&gt;

&lt;city&gt;Loops&lt;/city&gt;

&lt;state&gt;New York&lt;/state&gt;

&lt;zip&gt;01234&lt;/zip&gt;

&lt;/address&gt;

&lt;/student&gt;

&lt;student&gt;

&lt;id&gt;002&lt;/id&gt;

&lt;name&gt;Zoog&lt;/name&gt;

&lt;phone&gt;555-555-5555&lt;/phone&gt;

&lt;email&gt;zoog@planetzoron.uni&lt;/email&gt;

&lt;address&gt;

&lt;street&gt;45.3 Nebula 5&lt;/street&gt;

&lt;city&gt;Boolean City&lt;/city&gt;

&lt;state&gt;Booles&lt;/state&gt;

&lt;zip&gt;12358&lt;/zip&gt;

&lt;/address&gt;

&lt;/student&gt;

&lt;/students&gt;

Note the similarities to object-oriented programming. You can think of the XML tree in the following terms. The XML document represents an array of student objects. Each student object has multiple pieces of information, an ID, a name, a phone number, an email address, and a mailing address. The mailing address is also an object that has multiple pieces of data, such as street, city, state, and zip.

</p>

<p>

Let's look at some data made available from a web service such as Yahoo Weather. Here is the raw XML source. (Note I have edited it slightly for simplification purposes.)<br />

&lt;?xml version="1.0" encoding="UTF-8" standalone="yes" ?&gt;

&lt;rss version="2.0" xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"&gt;

&lt;channel&gt;

&lt;item&gt;

&lt;title&gt;Conditions for New York, NY at 12:49 pm EDT&lt;/title&gt;

&lt;geo:lat&gt;40.67&lt;/geo:lat&gt;

&lt;geo:long&gt;-73.94&lt;/geo:long&gt;

&lt;link&gt;http://us.rd.yahoo.com/dailynews/rss/weather/New_York__NY//link&gt;

&lt;pubDate&gt;Thu, 24 Jul 2014 12:49 pm EDT&lt;/pubDate&gt;

&lt;yweather:condition text="Partly Cloudy" code="30" temp="76"/&gt;

&lt;yweather:forecast day="Thu" low="65" high="82" text="Partly Cloudy"/&gt;

&lt;/item&gt;

&lt;/channel&gt;

&lt;/rss&gt;

The data is mapped in the tree stucture shown below:

</p>

<img src="imgs/data_07_weatherxml.jpg" class="tut" />

<p>

You may be wondering what the top level “RSS” is all about. Yahoo’s XML weather data is provided in RSS format. RSS stands for “Really Simple Syndication” and is a standardized XML format for syndicating web content (such as news articles, etc.). You can read more about <a href="http://en.wikipedia.org/wiki/RSS">RSS on Wikipedia</a>.

</p>

<p>Now that you have a handle on the tree structure, let's look at the specifics inside that structure. With the exception of the first line (which simply indicates that this page is XML formatted), this XML document contains a nested list of <em>elements</em>, each with a start tag, that is, <tt>&lt;channel&gt;</tt>, and an end tag, that is, <tt>&lt;/channel&gt;</tt>. Some of these elements have content between the tags:<br />

&lt;title&gt;Conditions for New York, NY at 12:49 pm EDT&lt;/title&gt;

and some have attributes (formatted by <em>Attribute Name</em> equals <em>Attribute Value in quotes</em>):<br />

&lt;yweather:forecast day="Thu" low="65" high="82" text="Partly Cloudy"/&gt;

<h3>Using the Processing XML Class</h3>

<p>

Since the syntax of XML is standardized, I could certainly use <tt>split()</tt>, <tt>indexof()</tt>, and <tt>substring()</tt> to find the pieces I want in the XML source. The point here, however, is that because XML is a standard format, I don't have to do this. Rather, I can use an XML parser. In Processing, XML can be parsed using the built-in Processing class <tt>XML</tt>.<br />

XML xml = loadXML("http://xml.weather.yahoo.com/forecastrss?p=10003");

Here, instead of <tt>loadStrings()</tt> or <tt>loadTable()</tt>, I'm now calling <tt>loadXML()</tt> and passing in the address (URL or local file) of the XML document. An <tt>XML</tt> object represents one element of an XML tree. When a document is first loaded, that <tt>XML</tt> object is always the root element. Referring back to XML tree diagram from Yahoo Weather, I find the current temperature via the following path:

</p>

<ol>

<li>

<p>The root of the tree is <strong>RSS</strong>.</p>

</li>

<li>

<p><strong>RSS</strong> has a child named <strong>channel</strong>.</p>

</li>

<li>

<p><strong>Channel</strong> has a child named <strong>item</strong>.</p>

</li>

<li>

<p><strong>Item</strong> has a child named <strong>yweather:condition</strong>.</p>

</li>

<li>

<p>The temperature is stored in <strong>yweather:condition</strong> as the attribute <strong>temp</strong>.</p>

</li>

</ol>

<p>

The children of an element are accessed via the <tt>getChild()</tt> function.<br />

// Accessing the “channel” child element of the root element --&gt;

XML channel = xml.getChild("channel");

The content of an element itself is retrieved with one of the following methods: <tt>getContent()</tt>, <tt>getIntContent()</tt>, or <tt>getFloatcContent()</tt>. <tt>getContent()</tt> is for generic use and will always give you the content as a String. If you intend to use the content as a number, Processing will convert it for you with either <tt>getIntContent()</tt> or <tt>getFloatcContent()</tt>. Attributes can also be read as either numbers — <tt>getInt(), getFloat()</tt> — or text — <tt>getString()</tt>.

</p>

<p>

Following steps one through five outlined above through the XML tree, I have:<br />

XML root = loadXML("http://xml.weather.yahoo.com/forecastrss?p=10003"); // Step 1

XML channel = root.getChild("channel"); // Step 2

XML item = channel.getChild("item"); // Step 3

XML yweather = item.getChild("yweather:condition"); // Step 4

int temperature = yweather.getInt("temp"); // Step 5

However, this is a bit long-winded and so can be compressed down into one (or three, as below) lines of code.<br />

XML root = loadXML(http://xml.weather.yahoo.com/forecastrss?p=10003);

XML forecast =

root.getChild("channel").getChild("item").getChild("yweather:condition"); // Steps 2-4

int temperature = forecast.getInt("temp");

Finally, the second line of code above can be further condensed to:<br />

XML forecast = xml.getChild("channel/item/yweather:condition"); // Steps 2-4

Following is the above code put together in an example that retrieves weather data for multiple zip codes by parsing Yahoo's XML feed.

</p>

<strong>Parsing Yahoo Weather XML</strong>

<img src="imgs/fig_18_17_weather.png" class="tut" />

int temperature = 0;Temperature is stored as a number and the description of the weather as a string.

String weather = "";

// The zip code

String zip = "10003";

void setup() {

size(200, 200);

// The URL for the XML document

String url = "http://xml.weather.yahoo.com/forecastrss?p=" + zip;

// Load the XML document

XML xml = loadXML(url);

// Here, I grab the XML element that I want.

XML forecast = xml.getChild("channel/item/yweather:forecast");

// And then I pull the attributes from that XML element.

temperature = forecast.getInt("high");

weather = forecast.getString("text");

}

void draw() {

background(255);

fill(0);

// Display all the stuff I want to display

text("Zip code: " + zip, 10, 50);

text("Today’s high: " + temperature, 10, 70);

text("Forecast: " + weather, 10, 90);

}

Other useful XML functions are:

</p>

<li>

<p><tt>hasChildren()</tt> — checks whether or not an element has any children</p>

</li>

<li>

<p><tt>getChildren()</tt> — returns an array containing all child elements</p>

</li>

<li>

<p><tt>getAttributeCount()</tt> — counts the specified element's number of attributes</p>

</li>

<li>

<p><tt>hasAttribute()</tt> — checks whether or not an element has the specified attribute</p>

</li>

<p>

In this example I'm accessing the child nodes by their name (i.e., "channel," "item,", etc.), however they can also be accessed numerically via an index (starting at zero, same as an array). This is convenient when looping over a list of children, much like I did with tabular data when I iterated over rows in a table.

</p>

<p>

Earlier, we used a <tt>Table</tt> to store information related to <tt>Bubble</tt> objects. XML document can also be used in the same manner. Here is a possible solution for an XML tree of <tt>Bubble</tt> objects:<br />

&lt;?xml version="1.0" encoding="UTF-8"?&gt;

&lt;bubbles&gt;

&lt;bubble&gt;

&lt;position x="160" y="103"/&gt;

&lt;diameter&gt;43.19838&lt;/diameter&gt;

&lt;label&gt;Happy&lt;/label&gt;

&lt;/bubble&gt;

&lt;bubble&gt;

&lt;position x="372" y="137"/&gt;

&lt;diameter&gt;52.42526&lt;/diameter&gt;

&lt;label&gt;Sad&lt;/label&gt;

&lt;/bubble&gt;

&lt;bubble&gt;

&lt;position x="273" y="235"/&gt;

&lt;diameter&gt;61.14072&lt;/diameter&gt;

&lt;label&gt;Joyous&lt;/label&gt;

&lt;/bubble&gt;

&lt;bubble&gt;

&lt;position x="121" y="179"/&gt;

&lt;diameter&gt;44.758068&lt;/diameter&gt;

&lt;label&gt;Melancholy&lt;/label&gt;

&lt;/bubble&gt;

&lt;/bubbles&gt;