andrew brooks

Sleeping Giant Rural Postman Problem

2017-12-01T00:00:00+00:00

This problem originated from a blog post I wrote for DataCamp on graph optimization here. The algorithm I sketched out there for solving the Chinese Problem on the Sleeping Giant state park trail network has since been formalized into the postman_problems python library. I’ve also added the Rural Postman solver that is implemented here.

So the three main enhancements in this post from the original DataCamp article and my second iteration published here updating to networkx 2.0 are:

OpenStreetMap for graph data and visualization.
Implementing the Rural Postman algorithm to consider optional edges.
Leveraging the postman_problems library.

This code, notebook and data for this post can be found in the postman_problems_examples repo.

The motivation and background around this problem is written up more thoroughly in the previous posts and postman_problems.

RPP Solution Route Animation

Here’s the full route animation. More details here. Kudos to my sister @laurabrooks for coding this up!

Create Graph from OSM
Viz Sleeping Giant Trails
Connect Edges
Viz Connected Component
Viz Trail Color
Contract Edges
Solve CPP
Solve RPP
Viz RPP Solution
- Viz: RPP optional edges
- Viz: RPP edges counts
Create geojson solution

import mplleaflet
import networkx as nx
import pandas as pd
import matplotlib.pyplot as plt
from collections import Counter

# can be found in https://github.com/brooksandrew/postman_problems_examples
from osm2nx import read_osm, haversine
from graph import contract_edges, create_rpp_edgelist

from postman_problems.tests.utils import create_mock_csv_from_dataframe
from postman_problems.solver import rpp, cpp
from postman_problems.stats import calculate_postman_solution_stats

Create Graph from OSM

# load OSM to a directed NX
g_d = read_osm('sleepinggiant.osm')  

# create an undirected graph
g = g_d.to_undirected()

<class 'networkx.classes.digraph.DiGraph'>

Adding edges that don’t exist on OSM, but should

g.add_edge('2318082790', '2318082832', id='white_horseshoe_fix_1')

Adding distance to OSM graph

Using the haversine formula to calculate distance between each edge.

for e in g.edges(data=True):
    e[2]['distance'] = haversine(g.node[e[0]]['lon'], 
                                 g.node[e[0]]['lat'], 
                                 g.node[e[1]]['lon'], 
                                 g.node[e[1]]['lat'])

Create graph of required trails only

A simple heuristic with a couple tweaks is all we need to create the graph with required edges:

Keep any edge with ‘Trail’ in the name attribute.
Manually remove the handful of trails that are not part of the required Giant Master route.

g_t = g.copy()

for e in g.edges(data=True):
    
    # remove non trails
    name = e[2]['name'] if 'name' in e[2] else ''
    if ('Trail' not in name.split()) or (name is None):
        g_t.remove_edge(e[0], e[1])
        
    # remove non Sleeping Giant trails
    elif name in [
        'Farmington Canal Linear Trail', 
        'Farmington Canal Heritage Trail', 
        'Montowese Trail',
        '(white blazes)']:
        g_t.remove_edge(e[0], e[1])

# cleaning up nodes left without edges
for n in nx.isolates(g_t.copy()):
    g_t.remove_node(n)

Viz Sleeping Giant Trails

All trails required for the Giant Master:

fig, ax = plt.subplots(figsize=(1,8))

pos = {k: (g_t.node[k]['lon'], g_t.node[k]['lat']) for k in g_t.nodes()}    
nx.draw_networkx_edges(g_t, pos, width=2.5, edge_color='black', alpha=0.7)

mplleaflet.save_html(fig, 'maps/sleepinggiant_trails_only.html')

Connect Edges

In order to run the RPP algorithm from postman_problems, the required edges of the graph must form a single connected component. We’re almost there with the Sleeping Giant trail map as-is, so we’ll just connect a few components manually.

Here’s an example of a few floating components (southwest corner of park):

OpenStreetMap makes finding these edge (way) IDs simple. Once grabbing the ? cursor, you can click on any edge to retrieve IDs and attributes.

Define OSM edges to add and remove from graph

edge_ids_to_add = [
    '223082783', 
    '223077827', 
    '40636272', 
    '223082785', 
    '222868698',
    '223083721',
    '222947116',
    '222711152',
    '222711155',
    '222860964',
    '223083718',
    '222867540',
    'white_horseshoe_fix_1'
]

edge_ids_to_remove = [
    '17220599'
]

Add attributes for supplementary edges

for e in g.edges(data=True):
    way_id = e[2].get('id').split('-')[0]
    if way_id in edge_ids_to_add:
        g_t.add_edge(e[0], e[1], **e[2])
        g_t.add_node(e[0], lat=g.node[e[0]]['lat'], lon=g.node[e[0]]['lon'])
        g_t.add_node(e[1], lat=g.node[e[1]]['lat'], lon=g.node[e[1]]['lon'])
    if way_id in edge_ids_to_remove:
        if g_t.has_edge(e[0], e[1]):
            g_t.remove_edge(e[0], e[1])
            
for n in nx.isolates(g_t.copy()):
    g_t.remove_node(n)

Ensuring that we’re left with one single connected component:

len(list(nx.connected_components(g_t)))

Viz Connected Component

The map below visualizes the required edges and nodes of interest (intersections and dead-ends where degree != 2):

fig, ax = plt.subplots(figsize=(1,12))

# edges
pos = {k: (g_t.node[k].get('lon'), g_t.node[k].get('lat')) for k in g_t.nodes()}    
nx.draw_networkx_edges(g_t, pos, width=3.0, edge_color='black', alpha=0.6)

# nodes (intersections and dead-ends)
pos_x = {k: (g_t.node[k]['lon'], g_t.node[k]['lat']) for k in g_t.nodes() if (g_t.degree(k)==1) | (g_t.degree(k)>2)}    
nx.draw_networkx_nodes(g_t, pos_x, nodelist=pos_x.keys(), node_size=35.0, node_color='red', alpha=0.9)

mplleaflet.save_html(fig, 'maps/trails_only_intersections.html')

Viz Trail Color

Because we can and it’s pretty.

name2color = {
    'Green Trail': 'green',
    'Quinnipiac Trail': 'blue',
    'Tower Trail': 'black',
    'Yellow Trail': 'yellow',
    'Red Square Trail': 'red',
    'White/Blue Trail Link': 'lightblue',
    'Orange Trail': 'orange',
    'Mount Carmel Avenue': 'black',
    'Violet Trail': 'violet',
    'blue Trail': 'blue',
    'Red Triangle Trail': 'red',
    'Blue Trail': 'blue',
    'Blue/Violet Trail Link': 'purple',
    'Red Circle Trail': 'red',
    'White Trail': 'gray',
    'Red Diamond Trail': 'red',
    'Yellow/Green Trail Link': 'yellowgreen',
    'Nature Trail': 'forestgreen',
    'Red Hexagon Trail': 'red',
    None: 'black'
}

fig, ax = plt.subplots(figsize=(1,10))
        
pos = {k: (g_t.node[k]['lon'], g_t.node[k]['lat']) for k in g_t.nodes()}   
e_color = [name2color[e[2].get('name')] for e in g_t.edges(data=True)]
nx.draw_networkx_edges(g_t, pos, width=3.0, edge_color=e_color, alpha=0.5)
nx.draw_networkx_nodes(g_t, pos_x, nodelist=pos_x.keys(), node_size=30.0, node_color='black', alpha=0.9)

mplleaflet.save_html(fig, 'maps/trails_only_color.html', tiles='cartodb_positron')

Check distance

This is strikingly close (within 0.25 miles) to what I calculated manually with some guess work from the SG trail map on the first pass at this problem here, before leveraging OSM.

print('{:0.2f} miles of required trail.'.format(sum([e[2]['distance']/1609.34 for e in g_t.edges(data=True)])))

25.56 miles of required trail.

Contract Edges

We could run the RPP algorithm on the graph as-is with >5000 edges. However, we can simplify computation by contracting edges into logical trail segments first. More details on the intuition and methodology in the 50 states post.

print('Number of edges in trail graph: {}'.format(len(g_t.edges())))

Number of edges in trail graph: 5141

# intialize contracted graph
g_tc = nx.MultiGraph()

# add contracted edges to graph
for ce in contract_edges(g_t, 'distance'):
    start_node, end_node, distance, path = ce
    
    contracted_edge = {
                'start_node': start_node,
                'end_node': end_node,
                'distance': distance,
                'name': g[path[0]][path[1]].get('name'),
                'required': 1,
                'path': path
            }
    
    g_tc.add_edge(start_node, end_node, **contracted_edge)
    g_tc.node[start_node]['lat'] = g.node[start_node]['lat']
    g_tc.node[start_node]['lon'] = g.node[start_node]['lon']
    g_tc.node[end_node]['lat'] = g.node[end_node]['lat']
    g_tc.node[end_node]['lon'] = g.node[end_node]['lon']

Edge contraction reduces the number of edges fed to the RPP algorithm by a factor of ~40.

print('Number of edges in contracted trail graoh: {}'.format(len(g_tc.edges())))

Number of edges in contracted trail graoh: 124

Solve CPP

First, let’s see how well the Chinese Postman solution works.

Create CPP edgelist

# create list with edge attributes and "from" & "to" nodes
tmp = []
for e in g_tc.edges(data=True):
    tmpi = e[2].copy()  # so we don't mess w original graph
    tmpi['start_node'] = e[0]
    tmpi['end_node'] = e[1]
    tmp.append(tmpi)
    
# create dataframe w node1 and node2 in order
eldf = pd.DataFrame(tmp)   
eldf = eldf[['start_node', 'end_node'] + list(set(eldf.columns)-{'start_node', 'end_node'})]

# create edgelist mock CSV
elfn = create_mock_csv_from_dataframe(eldf)

Start node

The route is designed to start at the far east end of the park on the Blue trail (node ‘735393342’). While the CPP and RPP solutions will return a Eulerian circuit (loop back to the starting node), we could truncate this last long doublebacking segment when actually running the route

Solve

circuit_cpp, gcpp = cpp(elfn, start_node='735393342')

CPP Stats

(distances in meters)

cpp_stats = calculate_postman_solution_stats(circuit_cpp)
cpp_stats

OrderedDict([('distance_walked', 54522.949121342645),
             ('distance_doublebacked', 13383.36715945256),
             ('distance_walked_once', 41139.581961890086),
             ('distance_walked_optional', 0),
             ('distance_walked_required', 54522.949121342645),
             ('edges_walked', 170),
             ('edges_doublebacked', 46),
             ('edges_walked_once', 124),
             ('edges_walked_optional', 0),
             ('edges_walked_required', 170)])

print('Miles in CPP solution: {:0.2f}'.format(cpp_stats['distance_walked']/1609.34))

Miles in CPP solution: 33.88

Solve RPP

With the CPP as benchmark, let’s see how well we do when we allow for optional edges in the route.

%%time
dfrpp = create_rpp_edgelist(g_tc, 
                            graph_full=g, 
                            edge_weight='distance', 
                            max_distance=2500)

CPU times: user 1min 39s, sys: 1.08 s, total: 1min 40s
Wall time: 1min 42s

Required vs optional edge counts

(1=required and 0=optional)

Counter( dfrpp['required'])

Counter({0: 3034, 1: 124})

Solve RPP

# create mockfilename
elfn = create_mock_csv_from_dataframe(dfrpp)

%%time
# solve
circuit_rpp, grpp = rpp(elfn, start_node='735393342')

CPU times: user 5.81 s, sys: 59.6 ms, total: 5.87 s
Wall time: 5.99 s

RPP Stats

(distances in meters)

rpp_stats = calculate_postman_solution_stats(circuit_rpp)
rpp_stats

OrderedDict([('distance_walked', 49427.7740637624),
             ('distance_doublebacked', 8288.19210187231),
             ('distance_walked_once', 41139.58196189009),
             ('distance_walked_optional', 5238.9032692701385),
             ('distance_walked_required', 44188.870794492264),
             ('edges_walked', 152),
             ('edges_doublebacked', 28),
             ('edges_walked_once', 124),
             ('edges_walked_optional', 12),
             ('edges_walked_required', 140)])

Leveraging the optional roads and trails, we’re able to shave about 3 miles off the CPP route. Total mileage checks in at 30.71, just under a 50K (30.1 miles).

print('Miles in RPP solution: {:0.2f}'.format(rpp_stats['distance_walked']/1609.34))

Miles in RPP solution: 30.71

Viz RPP Solution

# hack to convert 'path' from str back to list.  Caused by `create_mock_csv_from_dataframe`
for e in circuit_rpp:
    if type(e[3]['path']) == str:
        exec('e[3]["path"]=' + e[3]["path"])

Create graph from RPP solution

g_tcg = g_tc.copy()

# calc shortest path between optional nodes and add to graph
for e in circuit_rpp:
    granular_type = 'trail' if e[3]['required'] else 'optional'
    
    # add granular optional edges to g_tcg
    path = e[3]['path']
    for pair in list(zip(path[:-1], path[1:])):
        if (g_tcg.has_edge(pair[0], pair[1])) and (g_tcg[pair[0]][pair[1]][0].get('granular_type') == 'optional'):
                g_tcg[pair[0]][pair[1]][0]['granular_type'] = 'trail'
        else:
            g_tcg.add_edge(pair[0], pair[1], granular='True', granular_type=granular_type)
    
    # add granular nodes from optional edge paths to g_tcg
    for n in path:
        g_tcg.add_node(n, lat=g.node[n]['lat'], lon=g.node[n]['lon'])

Viz: RPP optional edges

The RPP algorithm picks up some logical shortcuts using the optional trails and a couple short stretches of road.

black: required trails
blue: optional trails and roads

fig, ax = plt.subplots(figsize=(1,8))

pos = {k: (g_tcg.node[k].get('lon'), g_tcg.node[k].get('lat')) for k in g_tcg.nodes()}    

el_opt = [e for e in g_tcg.edges(data=True) if e[2].get('granular_type') == 'optional'] 
nx.draw_networkx_edges(g_tcg, pos, edgelist=el_opt, width=6.0, edge_color='blue', alpha=1.0)

el_tr = [e for e in g_tcg.edges(data=True) if e[2].get('granular_type') == 'trail']
nx.draw_networkx_edges(g_tcg, pos, edgelist=el_tr, width=3.0, edge_color='black', alpha=0.8)

mplleaflet.save_html(fig, 'maps/rpp_solution_opt_edges.html', tiles='cartodb_positron')

Viz: RPP edges counts

## Create graph directly from rpp_circuit and original graph w lat/lon (g)
color_seq = [None, 'black', 'magenta', 'orange', 'yellow']
grppviz = nx.MultiGraph()

for e in circuit_rpp:
    for n1, n2 in zip(e[3]['path'][:-1], e[3]['path'][1:]):
        if grppviz.has_edge(n1, n2):
            grppviz[n1][n2][0]['linewidth'] += 2
            grppviz[n1][n2][0]['cnt'] += 1
        else:                
            grppviz.add_edge(n1, n2, linewidth=2.5)
            grppviz[n1][n2][0]['color_st'] = 'black' if g_t.has_edge(n1, n2) else 'red'
            grppviz[n1][n2][0]['cnt'] = 1
            grppviz.add_node(n1, lat=g.node[n1]['lat'], lon=g.node[n1]['lon'])
            grppviz.add_node(n2, lat=g.node[n2]['lat'], lon=g.node[n2]['lon']) 

for e in grppviz.edges(data=True):
    e[2]['color_cnt'] = color_seq[1] if 'cnt' not in e[2] else color_seq[e[2]['cnt'] ]

Edge walks per color:

black: 1
magenta: 2

fig, ax = plt.subplots(figsize=(1,10))

pos = {k: (grppviz.node[k]['lon'], grppviz.node[k]['lat']) for k in grppviz.nodes()}    
e_width = [e[2]['linewidth'] for e in grppviz.edges(data=True)]
e_color = [e[2]['color_cnt'] for e in grppviz.edges(data=True)]
nx.draw_networkx_edges(grppviz, pos, width=e_width, edge_color=e_color, alpha=0.7)

mplleaflet.save_html(fig, 'maps/rpp_solution_edge_cnts.html', tiles='cartodb_positron')

Create geojson solution

Used for the D3 route animation at the beginning of the post (also here). More details here.

geojson = {'features':[], 'type': 'FeatureCollection'}
time = 0
path = list(reversed(circuit_rpp[0][3]['path']))

for e in circuit_rpp:
    if e[3]['path'][0] != path[-1]: 
        path = list(reversed(e[3]['path']))
    else:
        path = e[3]['path']
    
    for n in path:
        time += 1
        doc = {'type': 'Feature',
              'properties': {
                  'latitude': g.node[n]['lat'],
                  'longitude': g.node[n]['lon'],
                  'time': time,
                  'id': e[3].get('id')
              },
              'geometry':{
                  'type': 'Point',
                  'coordinates': [g.node[n]['lon'], g.node[n]['lat']]
              }
          }
        geojson['features'].append(doc)

with open('circuit_rpp.geojson','w') as f:
    json.dump(geojson, f)

Sleeping Giant Rural Postman Problem was originally published by andrew brooks at andrew brooks on December 01, 2017.

50 states Rural Postman Problem

2017-11-19T00:00:00+00:00

Motivation

Recently I spent a considerable amount of time writing the postman_problems python library implementing solvers for the Chinese and Rural Postman Problems (CPP and RPP respectively). I wrote about my initial motivation for the project: finding the optimal route through a trail system in a state park here. Although I’ve still yet to run the 34 mile optimal trail route, I am pleased with the optimization procedure. However, I couldn’t help but feel that all those nights and weekends hobbying on this thing deserved a more satisfying visual than my static SVGs and hacky GIF. So to spice it up, I decided to solve the RPP on a graph derived from geodata and visualize on an interactive Leaflet map.

The Problem

In short, ride all 50 state named avenues in DC end-to-end following the shortest route possible.

There happens to be an annual 50 states ride sponsored by our regional bike association, WABA, that takes riders to each of the 50^† state named avenues in DC. Each state’s avenue is touched, but not covered in full. This problem takes it a step further by instituting this requirement. Thus, it boils to the RPP where the required edges are state avenues (end-to-end) and the optional edges are every other road within DC city limits.

For those unfamiliar with DC street naming convention, that can (and should) be remedied with a read through the history behind the street system here. Seriously, it’s an interesting read. Basically there are 50 state named avenues in DC ranging from 0.3 miles (Indiana Avenue) to 10 miles (Massachusetts Avenue) comprising 115 miles in total.

The Solution

The data is grabbed from Open Street Maps (OSM). Most of the post is spent wrangling the OSM geodata into shape for the RPP algorithm using NetworkX graphs. The final route (and intermediate steps) are visualized using Leaflet maps through mplleaflet which enable interactivity using tiles from Mapbox and CartoDB among others.

Note to readers: the rendering of these maps can work the browser pretty hard; allow a couple extra seconds for loading.

The Approach

Most of the heavy lifting leverages functions from the graph.py module in the postman_problems_examples repo. The majority of pre-RPP processing employs heuristics that simplify the computation such that this code can run in a reasonable amount of time. The parameters employed here, which I believe get pretty darn close to the optimal solution, run in about 50 minutes. By tweaking a couple parameters, accuracy can be sacrificed for time to get run time down to ~5 minutes on a standard 4 core laptop.

Verbose technical details about the guts of each step are omitted from this post for readability. However the interested reader can find these in the docstrings in graph.py.

The table of contents below provides the best high-level summary of the approach. All code needed to reproduce this analysis is in the postman_problems_examples repo, including the jupyter notebook used to author this blog post and a conda environment.

^† While there are 50 roadways, there are technically only 48 state named avenues: Ohio Drive and California Street are the stubborn exceptions.

0: Get the data
1: Load OSM to NetworkX
2: Make Graph w State Avenues only
- 2.1 Viz state avenues
3. Remove Redundant State Avenues
4. Create Single Connected Component
5. Solve CPP
6. Solve RPP

import mplleaflet
import networkx as nx
import pandas as pd
import matplotlib.pyplot as plt
from collections import Counter

# can be found in https://github.com/brooksandrew/postman_problems_examples
from osm2nx import read_osm, haversine
from graph import (
    states_to_state_avenue_name, subset_graph_by_edge_name, keep_oneway_edges_only, create_connected_components,
    create_unkinked_connected_components, nodewise_distance_connected_components,
    calculate_component_overlap, calculate_redundant_components, create_deduped_state_road_graph, 
    create_contracted_edge_graph, shortest_paths_between_components, find_minimum_weight_edges_to_connect_components,
    create_rpp_edgelist
    )

# can be found in https://github.com/brooksandrew/postman_problems
from postman_problems.tests.utils import create_mock_csv_from_dataframe
from postman_problems.solver import rpp, cpp
from postman_problems.stats import calculate_postman_solution_stats

0: Get the data

There are many ways to grab Open Street Map (OSM) data, since it’s, well, open. I grabbed the DC map from GeoFabrik here.

1: Load OSM to NetworkX

While some libraries like OSMnx provide an elegant interface to downloading, transforming and manipulating OSM data in NetworkX, I decided to start with the raw data itself. I adopted an OSM-to-nx parser from a hodge podge of Gists (here and there) to read_osm.

read_osm creates a directed graph. However, for this analysis, we’ll use undirected graphs with the assumption that all roads are bidirectional on a bike one way or another.

%%time

# load OSM to a directed NX
g = read_osm('district-of-columbia-latest.osm')  

# create an undirected graph
g_ud = g.to_undirected()

<class 'networkx.classes.digraph.DiGraph'>
CPU times: user 46.6 s, sys: 2.1 s, total: 48.7 s
Wall time: 50.2 s

This is a pretty big graph, about 275k edges. It takes about a minute to load on my machine (Macbook w 4 cores)

print(len(g.edges())) # number of edges

2: Make Graph w State Avenues only

Generate state avenue names

STATE_STREET_NAMES = [
    'Alabama','Alaska','Arizona','Arkansas','California','Colorado',
    'Connecticut','Delaware','Florida','Georgia','Hawaii','Idaho','Illinois',
    'Indiana','Iowa','Kansas','Kentucky','Louisiana','Maine','Maryland',
    'Massachusetts','Michigan','Minnesota','Mississippi','Missouri','Montana',
    'Nebraska','Nevada','New Hampshire','New Jersey','New Mexico','New York',
    'North Carolina','North Dakota','Ohio','Oklahoma','Oregon','Pennsylvania',
    'Rhode Island','South Carolina','South Dakota','Tennessee','Texas','Utah',
    'Vermont','Virginia','Washington','West Virginia','Wisconsin','Wyoming'
]

Most state avenues are written in the long form (ex. Connecticut Avenue Northwest). However, some, such as Florida Ave NW, are written in the short form. To be safe, we grab any permutation OSM could throw at us.

candidate_state_avenue_names = states_to_state_avenue_name(STATE_STREET_NAMES)

# two states break the "Avenue" pattern
candidate_state_avenue_names += ['California Street Northwest', 'Ohio Drive Southwest']

# preview
candidate_state_avenue_names[0:20]

['Alabama Ave Southeast',
 'Alabama Ave Southwest',
 'Alabama Ave Northeast',
 'Alabama Ave Northwest',
 'Alabama Ave SE',
 'Alabama Ave SW',
 'Alabama Ave NE',
 'Alabama Ave NW',
 'Alabama Avenue Southeast',
 'Alabama Avenue Southwest',
 'Alabama Avenue Northeast',
 'Alabama Avenue Northwest',
 'Alabama Avenue SE',
 'Alabama Avenue SW',
 'Alabama Avenue NE',
 'Alabama Avenue NW',
 'Alaska Ave Southeast',
 'Alaska Ave Southwest',
 'Alaska Ave Northeast',
 'Alaska Ave Northwest']

Create graph w state avenues only

g_st = subset_graph_by_edge_name(g, candidate_state_avenue_names)

# Add state edge attribute from full streetname (with avenue/drive and quandrant)
for e in g_st.edges(data=True):
    e[2]['state'] = e[2]['name'].rsplit(' ', 2)[0]

This is a much smaller graph:

print(len(g_st.edges()))

But every state is represented:

edge_count_by_state = Counter([e[2]['state'] for e in g_st.edges(data=True)])

# number of unique states 
print(len(edge_count_by_state))

Here they are by edge count:

edge_count_by_state

Counter({'Alabama': 348,
         'Alaska': 36,
         'Arizona': 89,
         'Arkansas': 64,
         'California': 24,
         'Colorado': 59,
         'Connecticut': 283,
         'Delaware': 66,
         'Florida': 253,
         'Georgia': 185,
         'Hawaii': 84,
         'Idaho': 50,
         'Illinois': 37,
         'Indiana': 29,
         'Iowa': 30,
         'Kansas': 74,
         'Kentucky': 47,
         'Louisiana': 37,
         'Maine': 199,
         'Maryland': 233,
         'Massachusetts': 581,
         'Michigan': 225,
         'Minnesota': 214,
         'Mississippi': 102,
         'Missouri': 66,
         'Montana': 74,
         'Nebraska': 183,
         'Nevada': 122,
         'New Hampshire': 259,
         'New Jersey': 170,
         'New Mexico': 66,
         'New York': 333,
         'North Carolina': 87,
         'North Dakota': 16,
         'Ohio': 391,
         'Oklahoma': 48,
         'Oregon': 170,
         'Pennsylvania': 433,
         'Rhode Island': 282,
         'South Carolina': 60,
         'South Dakota': 124,
         'Tennessee': 40,
         'Texas': 123,
         'Utah': 66,
         'Vermont': 154,
         'Virginia': 134,
         'Washington': 58,
         'West Virginia': 56,
         'Wisconsin': 257,
         'Wyoming': 27})

2.1 Viz state avenues

As long as your NetworkX graph has lat and lon node attributes, mplleaflet can be used to pretty effortlessly plot your NetworkX graph on an interactive map.

Here’s the map with all the state avenues…

fig, ax = plt.subplots(figsize=(1,8))

pos = {k: (g_st.node[k]['lon'], g_st.node[k]['lat']) for k in g_st.nodes()}    
nx.draw_networkx_edges(g_st, pos, width=4.0, edge_color='black', alpha=0.7)

# save viz    
mplleaflet.save_html(fig, 'state_avenues_all.html', tiles='cartodb_positron')

You can even customize with your favorite tiles. For example:

mplleaflet.display(fig=ax.figure, tiles='stamen_wc')

…But there’s a wrinkle. Zoom in on bigger avenues, like New York or Rhode Island, and you’ll notice that there are two parallel edges representing each direction as a separate one-way road. This usually occurs when there are several lanes of traffic in each direction, or physical dividers between directions. Example below:

This is great for OSM and point A to B routing problems, but for the Rural Postman problem it imposes the requirement that each main avenue be cycled twice. We’re not into that.

Example: Rhode Island Ave (parallel edges) vs Florida Ave (single edge)

3. Remove Redundant State Avenues

As it turns out, removing these parallel (redundant) edges is a nontrivial problem to solve. My approach is the following:

Build graph with one-way state avenue edges only.
For each state avenue, create list of connected components that represent sequences of OSM ways in the same direction (broken up by intersections and turns).
Compute distance between each node in a component to every other node in the other candidate components.
Identify redundant components as those with the majority of their nodes below some threshold distance away from another component.
Build graph without redundant edges.

3.1 Create state avenue graph with one-way edges only

g_st1 = keep_oneway_edges_only(g_st)

The one-way avenues are plotted in red below. A brief look indicates that 80-90% of the one-way avenues are parallel (redundant). A few, like Idaho Avenue NW and Ohio Drive SW, are single one-way roads with no accompanying parallel edge for us to remove.

NOTE: you’ll need to zoom in 3-4 levels to see the parallel edges.

fig, ax = plt.subplots(figsize=(1,6))

pos = {k: (g_st1.node[k]['lon'], g_st1.node[k]['lat']) for k in g_st1.nodes()}
nx.draw_networkx_edges(g_st1, pos, width=3.0, edge_color='red', alpha=0.7)

# save viz
mplleaflet.save_html(fig, 'oneway_state_avenues.html', tiles='cartodb_positron')

Create connected components with one-way state avenues

comps = create_connected_components(g_st1)

There are 163 distinct components in the graph above.

len(comps)

3.2 Split connected components

Remove kinked nodes

However, we need to break some of these components up into smaller ones. Many components, like the one below, have bends or a connected cycle that contain both the parallel edges, where we only want one. My approach is to identify the nodes with sharp angles and remove them. I don’t know what the proper name for these is (you can read about angular resolution), but we’ll call them “kinked nodes.”

This will split the connected component below into two, allowing us to determine that one of them is redundant.

I borrow this code from jeromer to calculate the compass bearing (0 to 360) of each edge. Wherever the the bearing difference between two adjacent edges is greater than bearing_thresh, we call the node shared by both edges a “kinked node.” A relative low bearing_thresh of 60 appeared to work best after some experimentation.

# create list of comps (graphs) without kinked nodes
comps_unkinked = create_unkinked_connected_components(comps=comps, bearing_thresh=60)

# comps in dict form for easy lookup
comps_dict = {comp.graph['id']:comp for comp in comps_unkinked}

After removing these “kinked nodes,” our list of components grows from 163 to 246:

len(comps_unkinked)

Viz components without kinked nodes

Example: Here’s the Massachusetts Ave example from above after we remove kinked nodes:

Full map: Zoom in on the map below and you’ll see that we split up most of the obvious components that should be. There are a few corner cases that we miss, but I’d estimate we programmatically split about 95% of the components correctly.

fig, ax = plt.subplots(figsize=(1,6))

for comp in comps_unkinked:
    pos = {k: (comp.node[k]['lon'], comp.node[k]['lat']) for k in comp.nodes()}
    nx.draw_networkx_edges(comp, pos, width=3.0, edge_color='orange', alpha=0.7)
    
mplleaflet.save_html(fig, 'oneway_state_avenues_without_kinked_nodes.html', tiles='cartodb_positron')

3.3 & 3.4 Match connected components

Now that we’ve crafted the right components, we calculate how close (parallel) each component is to one another.

This is a relatively coarse approach, but performs surprisingly well:

1. Find closest nodes from candidate components to each node in each component (pseudo code below):

  For each node N in component C:
    For each C_cand in components with same street avenue as C:
      Calculate closest node in C_cand to N.

2. Calculate overlap between components. Using the distances calculated in 1., we say that a node from component C is matched to a component C_cand if the distance is less than thresh_distance specified in calculate_component_overlap. 75 meters seemed to work pretty well. Essentially we’re saying these nodes are close enough to be considered interchangeable.

3. Use the node-wise matching calculated in 2. to calculate which components are redundant. If thresh_pct of nodes in component C are close enough (within thresh_distance) to nodes in component C_cand, we call C redundant and discard it.

# caclulate nodewise distances between each node in comp with closest node in each candidate
comp_matches = nodewise_distance_connected_components(comps_unkinked)

# calculate overlap between components
comp_overlap = calculate_component_overlap(comp_matches, thresh_distance=75)

# identify redundant and non-redundant components
remove_comp_ids, keep_comp_ids = calculate_redundant_components(comp_overlap, thresh_pct=0.75)

Viz redundant component solution

The map below visualizes the solution to the redundant parallel edges problem. There are some misses, but overall this simple approach works surprisingly well:

red: redundant one-way edges to remove
black: one-way edges to keep
blue: all state avenues

fig, ax = plt.subplots(figsize=(1,8))

# plot redundant one-way edges
for i, road in enumerate(remove_comp_ids):
    for comp_id in remove_comp_ids[road]:
        comp = comps_dict[comp_id]
        posc = {k: (comp.node[k]['lon'], comp.node[k]['lat']) for k in comp.nodes()}
        nx.draw_networkx_edges(comp, posc, width=7.0, edge_color='red')

# plot keeper one-way edges 
for i, road in enumerate(keep_comp_ids):
    for comp_id in keep_comp_ids[road]:
        comp = comps_dict[comp_id]
        posc = {k: (comp.node[k]['lon'], comp.node[k]['lat']) for k in comp.nodes()}
        nx.draw_networkx_edges(comp, posc, width=3.0, edge_color='black')

# plot all state avenues
pos_st = {k: (g_st.node[k]['lon'], g_st.node[k]['lat']) for k in g_st.nodes()}    
nx.draw_networkx_edges(g_st, pos_st, width=1.0, edge_color='blue', alpha=0.7)

mplleaflet.save_html(fig, 'redundant_edges.html', tiles='cartodb_positron')

3.5 Build graph without redundant edges

This is the essentially the graph with just black and blue edges from the map above.

# create a single graph with deduped state roads
g_st_nd = create_deduped_state_road_graph(g_st, comps_dict, remove_comp_ids)

After deduping the redundant edges, our connected component count drops from 246 to 96.

len(list(nx.connected_components(g_st_nd)))

4. Create Single Connected Component

The strategy I employ for solving the Rural Postman Problem (RPP) in postman_problems is simple in that it reuses the machinery from the Chinese Postman Problem (CPP) solver here. However, it makes the strong assumption that the graph’s required edges form a single connected component. This is obviously not true for our state avenue graph as-is, but it’s not too off. Although there are 96 components, there are only a couple more than a few hundred meters to the next closest component.

So we hack it a bit by adding required edges to the graph to make it a single connected component. The tricky part is choosing the edges that add as little distance as possible. This was the first computationally intensive step that required some clever tricks and approximations to ensure execution in a reasonable amount of time.

My approach:

Build graph with contracted edges only.
Calculate haversine distance between each possible pair of components.
Find minimum distance connectors: iterate through the data structure created in 2. to calculate shortest paths for top candidates based on haversine distance and add shortest connectors to graph. More details below.
Build single component graph.

4.1 Contract edges

Nodes with degree 2 are collapsed into an edge stretching from a dead-end node (degree 1) or intersection (degree >= 3) to another. This achieves two things:

Limits the number of distance calculations.
Ensures that components are connected at logical points (dead ends and intersections) rather than arbitrary parts of a roadway. This will make for a more continuous route.

# Create graph with contracted edges only
g_st_contracted = create_contracted_edge_graph(graph=g_st_nd, 
                                               edge_weight='length')

This significantly reduces the nodes needed for distances computations by a factor of > 15.

print('Number of nodes in contracted graph: {}'.format(len(g_st_contracted.nodes())))
print('Number of nodes in original graph: {}'.format(len(g_st_nd.nodes())))

Number of nodes in contracted graph: 347
Number of nodes in original graph: 5962

4.2 Calculate haversine distance between components

The 345 nodes from the contracted edge graph translate to >100,000 possible node pairings. That means >100,000 distance calculations. While applying a shortest path algorithm over the graph would certainly be more exact, it is painfully slow compared to simple haversine distance. This is mainly due to the high number of nodes and edges in the DC OSM map (over 250k edges).

On my laptop I averaged about 4 shortest path calculations per second. Not too bad for a handful, but 115k would take about 7 hours. Haversine distance, by comparison, churns through 115k in a couple seconds.

# create dataframe with shortest paths (haversine distance) between each component
dfsp = shortest_paths_between_components(g_st_contracted)

dfsp.shape[0]  # number of rows (node pairs)

4.3 Find minimum distance connectors

This gets a bit tricky. Basically we iterate through the top (closest) candidate pairs of components and connect them iteration-by-iteration with the shortest path edge. We use pre-calculated haversine distance to get in the right ballpark, then refine with true shortest path for the closest 20 candidates. This helps us avoid the scenario where we naively connect two nodes that are geographically close as the crow flies (haversine), but far away via available roads. Two nodes separated by highways or train tracks, for example.

# min weight edges that create a single connected component
connector_edges = find_minimum_weight_edges_to_connect_components(dfsp=dfsp, 
                                                                  graph=g_ud, 
                                                                  edge_weight='length', 
                                                                  top=20)

We had 96 components to connect, so it makes sense that we have 95 connectors.

len(connector_edges)

4.4 Build single component graph

# adding connector edges to create one single connected component
for e in connector_edges:
    g_st_contracted.add_edge(e[0], e[1], distance=e[2]['distance'], path=e[2]['path'], required=1, connector=True)

We add about 12 miles with the 95 additional required edges. That’s not too bad: an average distance of 0.13 miles per each edge added.

print(sum([e[2]['distance'] for e in g_st_contracted.edges(data=True) if e[2].get('connector')])/1609.34)

12.341087779659958

So that leaves us with a single component of 125 miles of required edges to optimize a route through. That means the distance of deduped state avenues alone, without connectors (~112 miles) is just a couple miles away from what Wikipedia reports (115 miles).

print(sum([e[2]['distance'] for e in g_st_contracted.edges(data=True)])/1609.34)

125.06645388481837

Make graph with granular edges (filling in those that were contracted) connecting components:

g1comp = g_st_contracted.copy()
for e in g_st_contracted.edges(data=True):
    if 'path' in e[2]:
        granular_type = 'connector' if 'connector' in e[2] else 'state'
        
        # add granular connector edges to graph 
        for pair in list(zip(e[2]['path'][:-1], e[2]['path'][1:])):
            g1comp.add_edge(pair[0], pair[1], granular='True', granular_type=granular_type)
            
        # add granular connector nodes to graph
        for n in e[2]['path']:
            g1comp.add_node(n, lat=g.node[n]['lat'], lon=g.node[n]['lon'])

4.5 Viz single connected component

Black edges represent the deduped state avenues. Red edges represent the 12 miles of connectors that create the single connected component.

fig, ax = plt.subplots(figsize=(1,6))

g1comp_conn = g1comp.copy()
g1comp_st = g1comp.copy()

for e in g1comp.edges(data=True):
    if ('granular_type' not in e[2]) or (e[2]['granular_type'] != 'connector'):
        g1comp_conn.remove_edge(e[0], e[1])

for e in g1comp.edges(data=True):
    if ('granular_type' not in e[2]) or (e[2]['granular_type'] != 'state'):
        g1comp_st.remove_edge(e[0], e[1])
    
pos = {k: (g1comp_conn.node[k]['lon'], g1comp_conn.node[k]['lat']) for k in g1comp_conn.nodes()}    
nx.draw_networkx_edges(g1comp_conn, pos, width=5.0, edge_color='red')

pos_st = {k: (g1comp_st.node[k]['lon'], g1comp_st.node[k]['lat']) for k in g1comp_st.nodes()}    
nx.draw_networkx_edges(g1comp_st, pos_st, width=3.0, edge_color='black')

# save viz
mplleaflet.save_html(fig, 'single_connected_comp.html', tiles='cartodb_positron')

5. Solve CPP

I don’t expect the Chinese Postman solution to be optimal since it only utilizes the required edges. However, I do expect it to execute quickly and serve as a benchmark for the Rural Postman solution. In the age of “deep learning,” I agree with Smerity, baselines need more love.

5.1 Create CPP edgelist

The cpp solver I wrote operates off an edgelist (text file). This feels a bit clunky here, but it works.

# create list with edge attributes and "from" & "to" nodes
tmp = []
for e in g_st_contracted.edges(data=True):
    tmpi = e[2].copy()  # so we don't mess w original graph
    tmpi['start_node'] = e[0]
    tmpi['end_node'] = e[1]
    tmp.append(tmpi)

# create dataframe w node1 and node2 in order
eldf = pd.DataFrame(tmp)   
eldf = eldf[['start_node', 'end_node'] + list(set(eldf.columns)-{'start_node', 'end_node'})]

The first two columns are interpeted as the from and to nodes; everything else as edge attributes.

eldf.head(3)

	start_node	end_node	comp	name	connector	path	distance	required
0	3788079770	649840206	0.0	Texas Avenue Southeast	NaN	[649840206, 649840209, 649840220, 30100500, 64...	1244.893849	1
1	3788079770	49744479	NaN	NaN	True	[3788079770, 49751669, 4630443674, 49751671, 4...	674.458290	1
2	49765126	49765287	1.0	Georgia Avenue Northwest	NaN	[49765126, 49765129, 49765130, 49765131, 49765...	559.251509	1

5.2 CPP solver

Starting point

I fix the starting node for the solution to OSM node 49765113 which corresponds to (38.917002, -77.0364987): the intersection of New Hampshire Avenue NW, 16th St NW and U St NW… and also the close to my house:

Solve

# create mockfilename
elfn = create_mock_csv_from_dataframe(eldf)

# solve
START_NODE = '49765113'  # New Hampshire Ave NW & U St NW.
circuit_cpp, gcpp = cpp(elfn, start_node=START_NODE)

5.3: CPP results

The CPP solution covers roughly 390,000 meters, about 242 miles. The optimal CPP route doubles the required distance, doublebacking every edge on average… definitely not ideal.

# circuit stats
calculate_postman_solution_stats(circuit_cpp)

OrderedDict([('distance_walked', 391523.723281576),
             ('distance_doublebacked', 190249.27638658223),
             ('distance_walked_once', 201274.44689499377),
             ('distance_walked_optional', 0),
             ('distance_walked_required', 391523.723281576),
             ('edges_walked', 688),
             ('edges_doublebacked', 337),
             ('edges_walked_once', 351),
             ('edges_walked_optional', 0),
             ('edges_walked_required', 688)])

6. Solve RPP

The RPP should improve the CPP solution as it considers optional edges that can drastically limit the amount of doublebacking.

We could add every possible edge that connects the required nodes, but it turns out that computation blows up quickly, and I’m not that patient. The get_shortest_paths_distances is the bottleneck applying dijkstra path length on all possible combinations. There are ~14k pairs to calculate shortest path for (4 per second) which would take almost one hour.

However, we can use some heuristics to speed this up dramatically without sacrificing too much.

6.1 Create RPP edgelist

Ideally optional edges will be relatively short, since they are, well, optional. It is unlikely that the RPP algorithm will find that leveraging an optional edge that stretches from one corner of the graph to another will be efficient. Thus we constrain the set of optional edges presented to the RPP solver to include only those less than max_distance.

I experimented with several thresholds. 3200 meters certainly took longer (~40 minutes), but yielded the best route results. I tried 4000m which ran for about 4 hours and returned a route with the same distance (160 miles) as the 3200m threshold.

%%time
dfrpp = create_rpp_edgelist(g_st_contracted=g_st_contracted, 
                            graph_full=g_ud, 
                            edge_weight='length', 
                            max_distance=3200)

CPU times: user 34min 38s, sys: 20.5 s, total: 34min 59s
Wall time: 36min 3s

Check how many optional edges are considered (0=optional, 1=required):

Counter(dfrpp['required'])

Counter({0: 16021, 1: 351})

6.2 RPP solver

Apply the RPP solver to the processed dataset.

%%time

# create mockfilename
elfn = create_mock_csv_from_dataframe(dfrpp)

# solve
circuit_rpp, grpp = rpp(elfn, start_node=START_NODE)

CPU times: user 7min 25s, sys: 1.68 s, total: 7min 27s
Wall time: 7min 31s

6.3 RPP results

As expected, the RPP route is considerably shorter than the CPP solution. The ~242 mile CPP route is cut significantly to ~160 with the RPP approach.

~26,000m (~161 miles) in total with ~59,000m (37 miles) of doublebacking. Not bad… but probably a 2-day ride.

# RPP route distance (miles)
print(sum([e[3]['distance'] for e in circuit_rpp])/1609.34)

161.58546891004886

# hack to convert 'path' from str back to list.  Caused by `create_mock_csv_from_dataframe`
for e in circuit_rpp:
    if type(e[3]['path']) == str:
        exec('e[3]["path"]=' + e[3]["path"])

calculate_postman_solution_stats(circuit_rpp)

OrderedDict([('distance_walked', 260045.958535698),
             ('distance_doublebacked', 58771.511640704295),
             ('distance_walked_once', 201274.4468949937),
             ('distance_walked_optional', 51891.485571184385),
             ('distance_walked_required', 208154.47296451364),
             ('edges_walked', 447),
             ('edges_doublebacked', 96),
             ('edges_walked_once', 351),
             ('edges_walked_optional', 57),
             ('edges_walked_required', 390)])

As seen below, filling the contracted edges back in with the granular nodes adds considerably to the edge count.

print('Number of edges in RPP circuit (with contracted edges): {}'.format(len(circuit_rpp)))
print('Number of edges in RPP circuit (with granular edges): {}'.format(rppdf.shape[0]))

Number of edges in RPP circuit (with contracted edges): 447
Number of edges in RPP circuit (with granular edges): 9081

6.4 Viz RPP graph

Create RPP granular graph

Add the granular edges (that we contracted for computation) back to the graph.

# calc shortest path between optional nodes and add to g1comp graph
for e in [e for e in circuit_rpp if e[3]['required']==0] :
    
    # add granular optional edges to g1comp
    path = e[3]['path']
    for pair in list(zip(path[:-1], path[1:])):
        if g1comp.has_edge(pair[0], pair[1]):
            continue
        g1comp.add_edge(pair[0], pair[1], granular='True', granular_type='optional')
    
    # add granular nodes from optional edge paths to g1comp
    for n in path:
        g1comp.add_node(n, lat=g.node[n]['lat'], lon=g.node[n]['lon'])

Visualize RPP solution by edge type

black: required state avenue edges
red: required non-state avenue edges added to form single component
blue: optional non-state avenue roads

fig, ax = plt.subplots(figsize=(1,12))

g1comp_conn = g1comp.copy()
g1comp_st = g1comp.copy()
g1comp_opt = g1comp.copy()

for e in g1comp.edges(data=True):
    if e[2].get('granular_type') != 'connector':
        g1comp_conn.remove_edge(e[0], e[1])

for e in g1comp.edges(data=True):
    #if e[2].get('name') not in candidate_state_avenue_names:
    if e[2].get('granular_type') != 'state':
        g1comp_st.remove_edge(e[0], e[1])

for e in g1comp.edges(data=True):
    if e[2].get('granular_type') != 'optional':
        g1comp_opt.remove_edge(e[0], e[1])

        
pos = {k: (g1comp_conn.node[k]['lon'], g1comp_conn.node[k]['lat']) for k in g1comp_conn.nodes()}    
nx.draw_networkx_edges(g1comp_conn, pos, width=6.0, edge_color='red')

pos_st = {k: (g1comp_st.node[k]['lon'], g1comp_st.node[k]['lat']) for k in g1comp_st.nodes()}    
nx.draw_networkx_edges(g1comp_st, pos_st, width=4.0, edge_color='black')

pos_opt = {k: (g1comp_opt.node[k]['lon'], g1comp_opt.node[k]['lat']) for k in g1comp_opt.nodes()}    
nx.draw_networkx_edges(g1comp_opt, pos_opt, width=2.0, edge_color='blue')

# save vbiz
mplleaflet.save_html(fig, 'rpp_solution_edge_type.html', tiles='cartodb_positron')

Visualize RPP solution by edge walk count

Edge walks per color:

black: 1
magenta: 2
orange: 3

Edges walked more than once are also widened.

This solution feels pretty reasonable with surprisingly little doublebacking. After staring at this for several minutes, I could think of roads I’d prefer not to cycle on, but no obvious shorter paths.

## Create graph directly from rpp_circuit and original graph w lat/lon (g_ud)
color_seq = [None, 'black', 'magenta', 'orange', 'yellow']
grppviz = nx.Graph()
edges_cnt = Counter([tuple(sorted([e[0], e[1]])) for e in circuit_rpp])

for e in circuit_rpp:
    for n1, n2 in zip(e[3]['path'][:-1], e[3]['path'][1:]):
        if grppviz.has_edge(n1, n2):
            grppviz[n1][n2]['linewidth'] += 2
            grppviz[n1][n2]['cnt'] += 1
        else:                
            grppviz.add_edge(n1, n2, linewidth=2.5)
            grppviz[n1][n2]['color_st'] = 'black' if g_st.has_edge(n1, n2) else 'red'
            grppviz[n1][n2]['cnt'] = 1
            grppviz.add_node(n1, lat=g_ud.node[n1]['lat'], lon=g_ud.node[n1]['lon'])
            grppviz.add_node(n2, lat=g_ud.node[n2]['lat'], lon=g_ud.node[n2]['lon']) 

for e in grppviz.edges(data=True):
    e[2]['color_cnt'] = color_seq[e[2]['cnt']]

fig, ax = plt.subplots(figsize=(1,12))

pos = {k: (grppviz.node[k]['lon'], grppviz.node[k]['lat']) for k in grppviz.nodes()}    
e_width = [e[2]['linewidth'] for e in grppviz.edges(data=True)]
e_color = [e[2]['color_cnt'] for e in grppviz.edges(data=True)]
nx.draw_networkx_edges(grppviz, pos, width=e_width, edge_color=e_color, alpha=0.7)
    
# save viz
mplleaflet.save_html(fig, 'rpp_solution_edge_cnt.html', tiles='cartodb_positron')

6.5 Serialize RPP solution

CSV

Remember we contracted the edges in 4.1 for more efficient computation. However, when we visualize the solution, the more granular edges within the larger contracted ones are filled back in, so we can see the exact route to ride with all the bends and squiggles.

# fill in RPP solution edgelist with granular nodes
rpplist = []
for ee in circuit_rpp:
    path = list(zip(ee[3]['path'][:-1], ee[3]['path'][1:]))
    for e in path:
        rpplist.append({
            'start_node': e[0],
            'end_node': e[1],
            'start_lat': g_ud.node[e[0]]['lat'],
            'start_lon': g_ud.node[e[0]]['lon'],
            'end_lat': g_ud.node[e[1]]['lat'],
            'end_lon': g_ud.node[e[1]]['lon'],
            'street_name': g_ud[e[0]][e[1]].get('name')
        })
        
# write solution to disk
rppdf = pd.DataFrame(rpplist)
rppdf.to_csv('rpp_solution.csv', index=False)

Geojson

Similarly, we create a geojson object of the RPP solution using the time attribute to keep track of the route order. This data structure can be used for fancy js/d3 visualizations. Coming soon, hopefully.

geojson = {'features':[], 'type': 'FeatureCollection'}
time = 0
path = list(reversed(circuit_rpp[0][3]['path']))

for e in circuit_rpp:
    if e[3]['path'][0] != path[-1]: 
        path = list(reversed(e[3]['path']))
    else:
        path = e[3]['path']
    
    for n in path:
        time += 1
        doc = {'type': 'Feature',
              'properties': {
                  'latitude': g.node[n]['lat'],
                  'longitude': g.node[n]['lon'],
                  'time': time,
                  'id': e[3].get('id')
              },
              'geometry':{
                  'type': 'Point',
                  'coordinates': [g.node[n]['lon'], g.node[n]['lat']]
              }
          }
        geojson['features'].append(doc)

with open('circuit_rpp.geojson','w') as f:
    json.dump(geojson, f)

50 states Rural Postman Problem was originally published by andrew brooks at andrew brooks on November 19, 2017.

Intro to graph optimization: solving the Chinese Postman Problem

2017-10-07T00:00:00+00:00

This post was originally published as a tutorial for DataCamp here on September 12 2017 using NetworkX 1.11. On September 20 2017, NetworkX announced the release of a new version 2.0, after two years in the making. While 2.0 introduces lots of great features (some have already been used to improve this project in postman_problems), it also introduced backwards incompatible API changes that broke the original tutorial :(. I’ve commented out lines deprecated by 2.0 and tagged with # deprecated after NX 1.11, so the changes made here are explicit. Most of the changes are around the passing and setting of attributes and return values deprecating lists for generators.

So… TL;DR:

This is the NetworkX 2.0 compatible version of the Chinese Postman DataCamp tutorial originally posted here.
The ideas introduced in this tutorial are packaged into the postman_problems library which is the mature implementation of these concepts.

A note on the making of this post. The original post was created in a Jupyter notebook and converted to HTML with some style tweaks by the DataCamp publishing team. This post was converted from an updated notebook to a Jekyll flavored markdown document for my blog using nb2jekyll with just a few tweaks of my own. This was the first Jupyter notebook I’ve converted to a blog post, but the conversion was smoother than I might have expected. I would recommend nb2jekyll and this post to comrade Jekyll bloggers looking to generate posts directly from Jupyter notebooks.

Intro to Graph Optimization with NetworkX in Python

Solving the Chinese Postman Problem

With this tutorial, you’ll tackle an established problem in graph theory called the Chinese Postman Problem. There are some components of the algorithm that while conceptually simple, turn out to be computationally rigorous. However, for this tutorial, only some prior knowledge of Python is required: no rigorous math, computer science or graph theory background is needed.

This tutorial will first go over the basic building blocks of graphs (nodes, edges, paths, etc) and solve the problem on a real graph (trail network of a state park) using the NetworkX library in Python. You’ll focus on the core concepts and implementation. For the interested reader, further reading on the guts of the optimization are provided.

Intro to Graph Optimization with NetworkX in Python
- Solving the Chinese Postman Problem
Motivating Graph Optimization
- The Problem
- Personal Motivation
Introducing Graphs
- NetworkX: Graph Manipulation and Analysis
- Installing Packages
Load Data
Create Graph
Inspect Graph
Visualize
- Manipulate Colors and Layout
- Plot
Overview of CPP Algorithm
Assumptions and Simplifications
CPP Step 1: Find Nodes of Odd Degree
CPP Step 2: Find Min Distance Pairs
CPP Step 3: Compute Eulerian Circuit
- Naive Circuit
- Correct Circuit
CPP Solution
- Text
- Stats
Visualize CPP Solution
Next Steps
References

Motivating Graph Optimization

The Problem

You’ve probably heard of the Travelling Salesman Problem which amounts to finding the shortest route (say, roads) that connects a set of nodes (say, cities). Although lesser known, the Chinese Postman Problem (CPP), also referred to as the Route Inspection or Arc Routing problem, is quite similar. The objective of the CPP is to find the shortest path that covers all the links (roads) on a graph at least once. If this is possible without doubling back on the same road twice, great; That’s the ideal scenario and the problem is quite simple. However, if some roads must be traversed more than once, you need some math to find the shortest route that hits every road at least once with the lowest total mileage.

Personal Motivation

(The following is a personal note: cheesy, cheeky and 100% not necessary for learning graph optimization in Python)

I had a real-life application for solving this problem: attaining the rank of Giantmaster Marathoner.

What is a Giantmaster? A Giantmaster is one (canine or human) who has hiked every trail of Sleeping Giant State Park in Hamden CT (neighbor to my hometown of Wallingford)… in their lifetime. A Giantmaster Marathoner is one who has hiked all these trails in a single day.

Thanks to the fastidious record keeping of the Sleeping Giant Park Association, the full roster of Giantmasters and their level of Giantmastering can be found here. I have to admit this motivated me quite a bit to kick-start this side-project and get out there to run the trails. While I myself achieved Giantmaster status in the winter of 2006 when I was a budding young volunteer of the Sleeping Giant Trail Crew (which I was pleased to see recorded in the SG archive), new challenges have since arisen. While the 12-month and 4-season Giantmaster categories are impressive and enticing, they’d also require more travel from my current home (DC) to my formative home (CT) than I could reasonably manage… and they’re not as interesting for graph optimization, so Giantmaster Marathon it is!

For another reference, the Sleeping Giant trail map is provided below:

Introducing Graphs

The nice thing about graphs is that the concepts and terminology are generally intuitive. Nonetheless, here’s some of the basic lingo:

Graphs are structures that map relations between objects. The objects are referred to as nodes and the connections between them as edges in this tutorial. Note that edges and nodes are commonly referred to by several names that generally mean exactly the same thing:

node == vertex == point
edge == arc == link

The starting graph is undirected. That is, your edges have no orientation: they are bi-directional. For example: A<--->B == B<--->A. By contrast, the graph you might create to specify the shortest path to hike every trail could be a directed graph, where the order and direction of edges matters. For example: A--->B != B--->A.

The graph is also an edge-weighted graph where the distance (in miles) between each pair of adjacent nodes represents the weight of an edge. This is handled as an edge attribute named “distance”.

Degree refers to the number of edges incident to (touching) a node. Nodes are referred to as odd-degree nodes when this number is odd and even-degree when even.

The solution to this CPP problem will be a Eulerian tour: a graph where a cycle that passes through every edge exactly once can be made from a starting node back to itself (without backtracking). An Euler Tour is also known by several names:

Eulerian tour == Eulerian circuit == Eulerian cycle

A matching is a subset of edges in which no node occurs more than once. A minimum weight matching finds the matching with the lowest possible summed edge weight.

NetworkX: Graph Manipulation and Analysis

NetworkX is the most popular Python package for manipulating and analyzing graphs. Several packages offer the same basic level of graph manipulation, notably igraph which also has bindings for R and C++. However, I found that NetworkX had the strongest graph algorithms that I needed to solve the CPP.

Installing Packages

If you’ve done any sort of data analysis in Python or have the Anaconda distribution, my guess is you probably have pandas and matplotlib. However, you might not have networkx. These should be the only dependencies outside the Python Standard Library that you’ll need to run through this tutorial. They are easy to install with pip:

pip install pandas
pip install networkx>=2.0
pip install matplotlib

These should be all the packages you’ll need for now. imageio and numpy are imported at the very end to create the GIF animation of the CPP solution. The animation is embedded within this post, so these packages are optional.

import itertools
import copy
import networkx as nx
import pandas as pd
import matplotlib.pyplot as plt

Load Data

Edge List

The edge list is a simple data structure that you’ll use to create the graph. Each row represents a single edge of the graph with some edge attributes.

node1 & node2: names of the nodes connected.
trail: edge attribute indicating the abbreviated name of the trail for each edge. For example: rs = red square
distance: edge attribute indicating trail length in miles.
color: trail color used for plotting.
estimate: edge attribute indicating whether the edge distance is estimated from eyeballing the trailmap (1=yes, 0=no) as some distances are not provided. This is solely for reference; it is not used for analysis.

# Grab edge list data hosted on Gist
edgelist = pd.read_csv('https://gist.githubusercontent.com/brooksandrew/e570c38bcc72a8d102422f2af836513b/raw/89c76b2563dbc0e88384719a35cba0dfc04cd522/edgelist_sleeping_giant.csv')

# Preview edgelist
edgelist.head(10)

	node1	node2	trail	distance	color
0	rs_end_north	v_rs	rs	0.30	red
1	v_rs	b_rs	rs	0.21	red
2	b_rs	g_rs	rs	0.11	red
3	g_rs	w_rs	rs	0.18	red
4	w_rs	o_rs	rs	0.21	red
5	o_rs	y_rs	rs	0.12	red
6	y_rs	rs_end_south	rs	0.39	red
7	rc_end_north	v_rc	rc	0.70	red
8	v_rc	b_rc	rc	0.04	red
9	b_rc	g_rc	rc	0.15	red

Node List

Node lists are usually optional in networkx and other graph libraries when edge lists are provided because the node names are provided in the edge list’s first two columns. However, in this case, there are some node attributes that we’d like to add: X, Y coordinates of the nodes (trail intersections) so that you can plot your graph with the same layout as the trail map.

I spent an afternoon annotating these manually by tracing over the image with GIMP:

id: name of the node corresponding to node1 and node2 in the edge list.
X: horizontal position/coordinate of the node relative to the topleft.
Y vertical position/coordinate of the node relative to the topleft.

Note on Generating the Node & Edge Lists

Creating the node names also took some manual effort. Each node represents an intersection of two or more trails. Where possible, the node is named by trail1_trail2 where trail1 precedes trail2 in alphabetical order.

Things got a little more difficult when the same trails intersected each other more than once. For example, the Orange and White trail. In these cases, I appended a _2 or _3 to the node name. For example, you have two distinct node names for the two distinct intersections of Orange and White: o_w and o_w_2.

This took a lot of trial and error and comparing the plots generated with X,Y coordinates to the real trail map.

# Grab node list data hosted on Gist
nodelist = pd.read_csv('https://gist.githubusercontent.com/brooksandrew/f989e10af17fb4c85b11409fea47895b/raw/a3a8da0fa5b094f1ca9d82e1642b384889ae16e8/nodelist_sleeping_giant.csv')

# Preview nodelist
nodelist.head(5)

	id	X	Y
0	b_bv	1486	732
1	b_bw	716	1357
2	b_end_east	3164	1111
3	b_end_west	141	1938
4	b_g	1725	771

Create Graph

Now you use the edge list and the node list to create a graph object in networkx.

# Create empty graph
g = nx.Graph()

Loop through the rows of the edge list and add each edge and its corresponding attributes to graph g.

# Add edges and edge attributes
for i, elrow in edgelist.iterrows():
    # g.add_edge(elrow[0], elrow[1], attr_dict=elrow[2:].to_dict())  # deprecated after NX 1.11
    g.add_edge(elrow[0], elrow[1], **elrow[2:].to_dict())

To illustrate what’s happening here, let’s print the values from the last row in the edge list that got added to graph g:

# Edge list example
print(elrow[0]) # node1
print(elrow[1]) # node2
print(elrow[2:].to_dict()) # edge attribute dict

o_gy2
y_gy2
{'estimate': 0, 'distance': 0.12, 'color': 'yellowgreen', 'trail': 'gy2'}

Similarly, you loop through the rows in the node list and add these node attributes.

# Add node attributes
for i, nlrow in nodelist.iterrows():
    # g.node[nlrow['id']] = nlrow[1:].to_dict()  # deprecated after NX 1.11
    nx.set_node_attributes(g, {nlrow['id']:  nlrow[1:].to_dict()})

Here’s an example from the last row of the node list:

# Node list example
print(nlrow)

id    y_rt
X      977
Y     1666
Name: 76, dtype: object

Inspect Graph

Edges

Your graph edges are represented by a list of tuples of length 3. The first two elements are the node names linked by the edge. The third is the dictionary of edge attributes.

# Preview first 5 edges

# g.edges(data=True)[0:5]  # deprecated after NX 1.11
list(g.edges(data=True))[0:5]

[('v_end_west',
  'b_v',
  {'color': 'violet', 'distance': 0.13, 'estimate': 0, 'trail': 'v'}),
 ('rt_end_north',
  'v_rt',
  {'color': 'red', 'distance': 0.3, 'estimate': 0, 'trail': 'rt'}),
 ('b_o',
  'park_east',
  {'color': 'orange', 'distance': 0.11, 'estimate': 0, 'trail': 'o'}),
 ('b_o',
  'o_gy2',
  {'color': 'orange', 'distance': 0.06, 'estimate': 0, 'trail': 'o'}),
 ('b_o',
  'b_y',
  {'color': 'blue', 'distance': 0.08, 'estimate': 0, 'trail': 'b'})]

Nodes

Similarly, your nodes are represented by a list of tuples of length 2. The first element is the node ID, followed by the dictionary of node attributes.

# Preview first 10 nodes

# g.nodes(data=True)[0:10]  # deprecated after NX 1.11
list(g.nodes(data=True))[0:10]

[('v_end_west', {'X': 359, 'Y': 1976}),
 ('rt_end_north', {'X': 681, 'Y': 850}),
 ('b_o', {'X': 2039, 'Y': 1012}),
 ('rh_end_north', {'X': 205, 'Y': 1472}),
 ('rh_end_tt_1', {'X': 558, 'Y': 1430}),
 ('o_y_tt_end_west', {'X': 459, 'Y': 1924}),
 ('w_rt', {'X': 926, 'Y': 1490}),
 ('b_rd_dupe', {'X': 268, 'Y': 1744}),
 ('b_tt_2', {'X': 857, 'Y': 1287}),
 ('rd_end_south_dupe', {'X': 273, 'Y': 1869})]

Summary Stats

Print out some summary statistics before visualizing the graph.

print('# of edges: {}'.format(g.number_of_edges()))
print('# of nodes: {}'.format(g.number_of_nodes()))

# of edges: 123
# of nodes: 77

Visualize

Manipulate Colors and Layout

Positions: First you need to manipulate the node positions from the graph into a dictionary. This will allow you to recreate the graph using the same layout as the actual trail map. Y is negated to transform the Y-axis origin from the topleft to the bottomleft.

# Define node positions data structure (dict) for plotting
node_positions = {node[0]: (node[1]['X'], -node[1]['Y']) for node in g.nodes(data=True)}

# Preview of node_positions with a bit of hack (there is no head/slice method for dictionaries).
dict(list(node_positions.items())[0:5])

{'b_o': (2039, -1012),
 'rh_end_north': (205, -1472),
 'rh_end_tt_1': (558, -1430),
 'rt_end_north': (681, -850),
 'v_end_west': (359, -1976)}

Colors: Now you manipulate the edge colors from the graph into a simple list so that you can visualize the trails by their color.

# Define data structure (list) of edge colors for plotting

# edge_colors = [e[2]['color'] for e in g.edges(data=True)]  # deprecated after NX 1.11
edge_colors = [e[2]['color'] for e in list(g.edges(data=True))]

# Preview first 10
edge_colors[0:10]

['violet',
 'red',
 'orange',
 'orange',
 'blue',
 'blue',
 'red',
 'red',
 'black',
 'black']

Plot

Now you can make a nice plot that lines up nicely with the Sleeping Giant trail map:

plt.figure(figsize=(8, 6))
nx.draw(g, pos=node_positions, edge_color=edge_colors, node_size=10, node_color='black')
plt.title('Graph Representation of Sleeping Giant Trail Map', size=15)
plt.show()

This graph representation obviously doesn’t capture all the trails’ bends and squiggles, however not to worry: these are accurately captured in the edge distance attribute which is used for computation. The visual does capture distance between nodes (trail intersections) as the crow flies, which appears to be a decent approximation.

Overview of CPP Algorithm

OK, so now that you’ve defined some terms and created the graph, how do you find the shortest path through it?

Solving the Chinese Postman Problem is quite simple conceptually:

Find all nodes with odd degree (very easy).
(Find all trail intersections where the number of trails touching that intersection is an odd number)
Add edges to the graph such that all nodes of odd degree are made even. These added edges must be duplicates from the original graph (we’ll assume no bushwhacking for this problem). The set of edges added should sum to the minimum distance possible (hard…np-hard to be precise).
(In simpler terms, minimize the amount of double backing on a route that hits every trail)
Given a starting point, find the Eulerian tour over the augmented dataset (moderately easy).
(Once we know which trails we’ll be double backing on, actually calculate the route from beginning to end)

Assumptions and Simplifications

While a shorter and more precise path could be generated by relaxing the assumptions below, this would add complexity beyond the scope of this tutorial which focuses on the CPP.

Assumption 1: Required trails only

As you can see from the trail map above, there are roads along the borders of the park that could be used to connect trails, particularly the red trails. There are also some trails (Horseshoe and unmarked blazes) which are not required per the Giantmaster log, but could be helpful to prevent lengthy double backing. The inclusion of optional trails is actually an established variant of the CPP called the Rural Postman Problem. We ignore optional trails in this tutorial and focus on required trails only.

Assumption 2: Uphill == downhill

The CPP assumes that the cost of walking a trail is equivalent to its distance, regardless of which direction it is walked. However, some of these trails are rather hilly and will require more energy to walk up than down. Some metric that combines both distance and elevation change over a directed graph could be incorporated into an extension of the CPP called the Windy Postman Problem.

Assumption 3: No parallel edges (trails)

While possible, the inclusion of parallel edges (multiple trails connecting the same two nodes) adds complexity to computation. Luckily this only occurs twice here (Blue <=> Red Diamond and Blue <=> Tower Trail). This is addressed by a bit of a hack to the edge list: duplicate nodes are included with a _dupe suffix to capture every trail while maintaining uniqueness in the edges. The CPP implementation in the postman_problems package I wrote robustly handles parallel edges in a more elegant way if you’d like to solve the CPP on your own graph with many parallel edges.

CPP Step 1: Find Nodes of Odd Degree

This is a pretty straightforward counting computation. You see that 36 of the 76 nodes have odd degree. These are mostly the dead-end trails (degree 1) and intersections of 3 trails. There are a handful of degree 5 nodes.

# Calculate list of nodes with odd degree
# nodes_odd_degree = [v for v, d in g.degree_iter() if d % 2 == 1]  # deprecated after NX 1.11
nodes_odd_degree = [v for v, d in g.degree() if d % 2 == 1]
        
# Preview
nodes_odd_degree[0:5]

['v_end_west',
 'rt_end_north',
 'rh_end_north',
 'rh_end_tt_1',
 'o_y_tt_end_west']

# Counts
print('Number of nodes of odd degree: {}'.format(len(nodes_odd_degree)))
print('Number of total nodes: {}'.format(len(g.nodes())))

Number of nodes of odd degree: 36
Number of total nodes: 77

CPP Step 2: Find Min Distance Pairs

This is really the meat of the problem. You’ll break it down into 5 parts:

Compute all possible pairs of odd degree nodes.
Compute the shortest path between each node pair calculated in 1.
Create a complete graph connecting every node pair in 1. with shortest path distance attributes calculated in 2.
Compute a minimum weight matching of the graph calculated in 3.
(This boils down to determining how to pair the odd nodes such that the sum of the distance between the pairs is as small as possible).
Augment the original graph with the shortest paths between the node pairs calculated in 4.

Step 2.1: Compute Node Pairs

You use the itertools combination function to compute all possible pairs of the odd degree nodes. Your graph is undirected, so we don’t care about order: For example, (a,b) == (b,a).

# Compute all pairs of odd nodes. in a list of tuples
odd_node_pairs = list(itertools.combinations(nodes_odd_degree, 2))

# Preview pairs of odd degree nodes
odd_node_pairs[0:10]

[('v_end_west', 'rt_end_north'),
 ('v_end_west', 'rh_end_north'),
 ('v_end_west', 'rh_end_tt_1'),
 ('v_end_west', 'o_y_tt_end_west'),
 ('v_end_west', 'b_v'),
 ('v_end_west', 'y_gy2'),
 ('v_end_west', 'nature_end_west'),
 ('v_end_west', 'y_rh'),
 ('v_end_west', 'g_gy2'),
 ('v_end_west', 'b_bv')]

# Counts
print('Number of pairs: {}'.format(len(odd_node_pairs)))

Number of pairs: 630

Let’s confirm that this number of pairs is correct with a the combinatoric below. Luckily, you only have 630 pairs to worry about. Your computation time to solve this CPP example is trivial (a couple seconds).

However, if you had 3,600 odd node pairs instead, you’d have ~6.5 million pairs to optimize. That’s a ~10,000x increase in output given a 100x increase in input size.

$\#\;of\;pairs = n\;choose\;r = {n \choose r} = \frac{n!}{r!(n-r)!} = \frac{36!}{2! (36-2)!} = 630$

Step 2.2: Compute Shortest Paths between Node Pairs

This is the first step that involves some real computation. Luckily networkx has a convenient implementation of Dijkstra’s algorithm to compute the shortest path between two nodes. You apply this function to every pair (all 630) calculated above in odd_node_pairs.

def get_shortest_paths_distances(graph, pairs, edge_weight_name):
    """Compute shortest distance between each pair of nodes in a graph.  Return a dictionary keyed on node pairs (tuples)."""
    distances = {}
    for pair in pairs:
        distances[pair] = nx.dijkstra_path_length(graph, pair[0], pair[1], weight=edge_weight_name)
    return distances

# Compute shortest paths.  Return a dictionary with node pairs keys and a single value equal to shortest path distance.
odd_node_pairs_shortest_paths = get_shortest_paths_distances(g, odd_node_pairs, 'distance')

# Preview with a bit of hack (there is no head/slice method for dictionaries).
dict(list(odd_node_pairs_shortest_paths.items())[0:10])

{('b_bv', 'rs_end_north'): 0.8999999999999999,
 ('b_v', 'rh_end_tt_3'): 0.6500000000000001,
 ('g_gy2', 'b_bw'): 1.73,
 ('o_y_tt_end_west', 'rh_end_tt_4'): 0.35,
 ('rd_end_north', 'g_gy1'): 1.37,
 ('rd_end_south', 'b_tt_3'): 1.1300000000000001,
 ('rd_end_south', 'g_gy1'): 1.3,
 ('rh_end_tt_1', 'rd_end_south'): 0.6000000000000001,
 ('rt_end_north', 'rd_end_south'): 1.31,
 ('v_end_west', 'b_end_west'): 0.45}

Step 2.3: Create Complete Graph

A complete graph is simply a graph where every node is connected to every other node by a unique edge.

Here’s a basic example from Wikipedia of a 7 node complete graph with 21 (7 choose 2) edges:

The graph you create below has 36 nodes and 630 edges with their corresponding edge weight (distance).

create_complete_graph is defined to calculate it. The flip_weights parameter is used to transform the distance to the weight attribute where smaller numbers reflect large distances and high numbers reflect short distances. This sounds a little counter intuitive, but is necessary for Step 2.4 where you calculate the minimum weight matching on the complete graph.

Ideally you’d calculate the minimum weight matching directly, but NetworkX only implements a max_weight_matching function which maximizes, rather than minimizes edge weight. We hack this a bit by negating (multiplying by -1) the distance attribute to get weight. This ensures that order and scale by distance are preserved, but reversed.

def create_complete_graph(pair_weights, flip_weights=True):
    """
    Create a completely connected graph using a list of vertex pairs and the shortest path distances between them
    Parameters: 
        pair_weights: list[tuple] from the output of get_shortest_paths_distances
        flip_weights: Boolean. Should we negate the edge attribute in pair_weights?
    """
    g = nx.Graph()
    for k, v in pair_weights.items():
        wt_i = - v if flip_weights else v
        # g.add_edge(k[0], k[1], {'distance': v, 'weight': wt_i})  # deprecated after NX 1.11 
        g.add_edge(k[0], k[1], **{'distance': v, 'weight': wt_i})  
    return g

# Generate the complete graph
g_odd_complete = create_complete_graph(odd_node_pairs_shortest_paths, flip_weights=True)

# Counts
print('Number of nodes: {}'.format(len(g_odd_complete.nodes())))
print('Number of edges: {}'.format(len(g_odd_complete.edges())))

Number of nodes: 36
Number of edges: 630

For a visual prop, the fully connected graph of odd degree node pairs is plotted below. Note that you preserve the X, Y coordinates of each node, but the edges do not necessarily represent actual trails. For example, two nodes could be connected by a single edge in this graph, but the shortest path between them could be 5 hops through even degree nodes (not shown here).

# Plot the complete graph of odd-degree nodes
plt.figure(figsize=(8, 6))
pos_random = nx.random_layout(g_odd_complete)
nx.draw_networkx_nodes(g_odd_complete, node_positions, node_size=20, node_color="red")
nx.draw_networkx_edges(g_odd_complete, node_positions, alpha=0.1)
plt.axis('off')
plt.title('Complete Graph of Odd-degree Nodes')
plt.show()

Step 2.4: Compute Minimum Weight Matching

This is the most complex step in the CPP. You need to find the odd degree node pairs whose combined sum (of distance between them) is as small as possible. So for your problem, this boils down to selecting the optimal 18 edges (36 odd degree nodes / 2) from the hairball of a graph generated in 2.3.

Both the implementation and intuition of this optimization are beyond the scope of this tutorial… like 800+ lines of code and a body of academic literature beyond this scope.

However, a quick aside for the interested reader:

A huge thanks to Joris van Rantwijk for writing the orginal implementation on his blog way back in 2008. I stumbled into the problem a similar way with the same intention as Joris. From Joris’s 2008 post:

Since I did not find any Perl implementations of maximum weighted matching, I lightly decided to write some code myself. It turned out that I had underestimated the problem, but by the time I realized my mistake, I was so obsessed with the problem that I refused to give up.

However, I did give up. Luckily Joris did not.

This Maximum Weight Matching has since been folded into and maintained within the NetworkX package. Another big thanks to the 10+ contributors on GitHub who have maintained this hefty codebase.

This is a hard and intensive computation. The first breakthrough in 1965 proved that the Maximum Matching problem could be solved in polynomial time. It was published by Jack Edmonds with perhaps one of the most beautiful academic paper titles ever: “Paths, trees, and flowers” [1]. A body of literature has since built upon this work, improving the optimization procedure. The code implemented in the NetworkX function max_weight_matching is based on Galil, Zvi (1986) [2] which employs an O(n³) time algorithm.

# Compute min weight matching.
# Note: max_weight_matching uses the 'weight' attribute by default as the attribute to maximize.
odd_matching_dupes = nx.algorithms.max_weight_matching(g_odd_complete, True)

print('Number of edges in matching: {}'.format(len(odd_matching_dupes)))

Number of edges in matching: 36

The matching output (odd_matching_dupes) is a dictionary. Although there are 36 edges in this matching, you only want 18. Each edge-pair occurs twice (once with node 1 as the key and a second time with node 2 as the key of the dictionary).

# Preview of matching with dupes
odd_matching_dupes

{'b_bv': 'v_bv',
 'b_bw': 'rh_end_tt_1',
 'b_end_east': 'g_gy2',
 'b_end_west': 'b_v',
 'b_tt_3': 'rt_end_north',
 'b_v': 'b_end_west',
 'g_gy1': 'rc_end_north',
 'g_gy2': 'b_end_east',
 'g_w': 'w_bw',
 'nature_end_west': 'o_y_tt_end_west',
 'o_rt': 'o_w_1',
 'o_tt': 'rh_end_tt_2',
 'o_w_1': 'o_rt',
 'o_y_tt_end_west': 'nature_end_west',
 'rc_end_north': 'g_gy1',
 'rc_end_south': 'y_gy1',
 'rd_end_north': 'rh_end_north',
 'rd_end_south': 'v_end_west',
 'rh_end_north': 'rd_end_north',
 'rh_end_south': 'y_rh',
 'rh_end_tt_1': 'b_bw',
 'rh_end_tt_2': 'o_tt',
 'rh_end_tt_3': 'rh_end_tt_4',
 'rh_end_tt_4': 'rh_end_tt_3',
 'rs_end_north': 'v_end_east',
 'rs_end_south': 'y_gy2',
 'rt_end_north': 'b_tt_3',
 'rt_end_south': 'y_rt',
 'v_bv': 'b_bv',
 'v_end_east': 'rs_end_north',
 'v_end_west': 'rd_end_south',
 'w_bw': 'g_w',
 'y_gy1': 'rc_end_south',
 'y_gy2': 'rs_end_south',
 'y_rh': 'rh_end_south',
 'y_rt': 'rt_end_south'}

You convert this dictionary to a list of tuples since you have an undirected graph and order does not matter. Removing duplicates yields the unique 18 edge-pairs that cumulatively sum to the least possible distance.

# Convert matching to list of deduped tuples
odd_matching = list(pd.unique([tuple(sorted([k, v])) for k, v in odd_matching_dupes.items()]))

# Counts
print('Number of edges in matching (deduped): {}'.format(len(odd_matching)))

Number of edges in matching (deduped): 18

# Preview of deduped matching
odd_matching

[('o_tt', 'rh_end_tt_2'),
 ('nature_end_west', 'o_y_tt_end_west'),
 ('b_tt_3', 'rt_end_north'),
 ('rs_end_south', 'y_gy2'),
 ('b_bw', 'rh_end_tt_1'),
 ('rd_end_south', 'v_end_west'),
 ('b_bv', 'v_bv'),
 ('b_end_west', 'b_v'),
 ('rh_end_south', 'y_rh'),
 ('g_gy1', 'rc_end_north'),
 ('rc_end_south', 'y_gy1'),
 ('rd_end_north', 'rh_end_north'),
 ('rt_end_south', 'y_rt'),
 ('rh_end_tt_3', 'rh_end_tt_4'),
 ('b_end_east', 'g_gy2'),
 ('rs_end_north', 'v_end_east'),
 ('g_w', 'w_bw'),
 ('o_rt', 'o_w_1')]

Let’s visualize these pairs on the complete graph plotted earlier in step 2.3. As before, while the node positions reflect the true graph (trail map) here, the edge distances shown (blue lines) are as the crow flies. The actual shortest route from one node to another could involve multiple edges that twist and turn with considerably longer distance.

plt.figure(figsize=(8, 6))

# Plot the complete graph of odd-degree nodes
nx.draw(g_odd_complete, pos=node_positions, node_size=20, alpha=0.05)

# Create a new graph to overlay on g_odd_complete with just the edges from the min weight matching
g_odd_complete_min_edges = nx.Graph(odd_matching)
nx.draw(g_odd_complete_min_edges, pos=node_positions, node_size=20, edge_color='blue', node_color='red')

plt.title('Min Weight Matching on Complete Graph')
plt.show()

To illustrate how this fits in with the original graph, you plot the same min weight pairs (blue lines), but over the trail map (faded) instead of the complete graph. Again, note that the blue lines are the bushwhacking route (as the crow flies edges, not actual trails). You still have a little bit of work to do to find the edges that comprise the shortest route between each pair in Step 3.

plt.figure(figsize=(8, 6))

# Plot the original trail map graph
nx.draw(g, pos=node_positions, node_size=20, alpha=0.1, node_color='black')

# Plot graph to overlay with just the edges from the min weight matching
nx.draw(g_odd_complete_min_edges, pos=node_positions, node_size=20, alpha=1, node_color='red', edge_color='blue')

plt.title('Min Weight Matching on Orginal Graph')
plt.show()

Step 2.5: Augment the Original Graph

Now you augment the original graph with the edges from the matching calculated in 2.4. A simple function to do this is defined below which also notes that these new edges came from the augmented graph. You’ll need to know this in ** 3.** when you actually create the Eulerian circuit through the graph.

def add_augmenting_path_to_graph(graph, min_weight_pairs):
    """
    Add the min weight matching edges to the original graph
    Parameters:
        graph: NetworkX graph (original graph from trailmap)
        min_weight_pairs: list[tuples] of node pairs from min weight matching
    Returns:
        augmented NetworkX graph
    """
    
    # We need to make the augmented graph a MultiGraph so we can add parallel edges
    graph_aug = nx.MultiGraph(graph.copy())
    for pair in min_weight_pairs:
        graph_aug.add_edge(pair[0], 
                           pair[1], 
                           **{'distance': nx.dijkstra_path_length(graph, pair[0], pair[1]), 'trail': 'augmented'}
                           # attr_dict={'distance': nx.dijkstra_path_length(graph, pair[0], pair[1]),
                           #            'trail': 'augmented'}  # deprecated after 1.11
                          )
    return graph_aug

Let’s confirm that your augmented graph adds the expected number (18) of edges:

# Create augmented graph: add the min weight matching edges to g
g_aug = add_augmenting_path_to_graph(g, odd_matching)

# Counts
print('Number of edges in original graph: {}'.format(len(g.edges())))
print('Number of edges in augmented graph: {}'.format(len(g_aug.edges())))

Number of edges in original graph: 123
Number of edges in augmented graph: 141

Let’s also confirm that every node now has even degree:

# pd.value_counts(g_aug.degree())  # deprecated after NX 1.11
pd.value_counts([e[1] for e in g_aug.degree()])

  54
  18
   5
dtype: int64

CPP Step 3: Compute Eulerian Circuit

Now that you have a graph with even degree the hard optimization work is over. As Euler famously postulated in 1736 with the Seven Bridges of Königsberg problem, there exists a path which visits each edge exactly once if all nodes have even degree. Carl Hierholzer fomally proved this result later in the 1870s.

There are many Eulerian circuits with the same distance that can be constructed. You can get 90% of the way there with the NetworkX eulerian_circuit function. However there are some limitations.

Limitations you will fix:

The augmented graph could (and likely will) contain edges that didn’t exist on the original graph. To get the circuit (without bushwhacking), you must break down these augmented edges into the shortest path through the edges that actually exist.
eulerian_circuit only returns the order in which we hit each node. It does not return the attributes of the edges needed to complete the circuit. This is necessary because you need to keep track of which edges have been walked already when multiple edges exist between two nodes.

Limitations you won’t fix:

To save your legs some work, you could relax the assumption of the Eulerian circuit that one start and finish at the same node. An [Eulerian path] (the general case of the Eulerian circuit), can also be found if there are exactly two nodes of odd degree. This would save you a little bit of double backing...presuming you could get a ride back from the other end of the park. However, at the time of this writing, NetworkX does not provide a Euler Path algorithm. The [eulerian_circuit code] isn't too bad and could be adopted for this case, but you'll keep it simple here.

Naive Circuit

Nonetheless, let’s start with the simple yet incomplete solution:

naive_euler_circuit = list(nx.eulerian_circuit(g_aug, source='b_end_east'))

As expected, the length of the naive Eulerian circuit is equal to the number of the edges in the augmented graph.

print('Length of eulerian circuit: {}'.format(len(naive_euler_circuit)))

Length of eulerian circuit: 141

The output is just a list of tuples which represent node pairs. Note that the first node of each pair is the same as the second node from the preceding pair.

# Preview naive Euler circuit
naive_euler_circuit[0:10]

[('b_end_east', 'b_y'),
 ('b_y', 'y_gy2'),
 ('y_gy2', 'rs_end_south'),
 ('rs_end_south', 'y_rs'),
 ('y_rs', 'y_gy2'),
 ('y_gy2', 'o_gy2'),
 ('o_gy2', 'o_rs'),
 ('o_rs', 'o_w_2'),
 ('o_w_2', 'w_rc'),
 ('w_rc', 'y_rc')]

Correct Circuit

Now let’s define a function that utilizes the original graph to tell you which trails to use to get from node A to node B. Although verbose in code, this logic is actually quite simple. You simply transform the naive circuit which included edges that did not exist in the original graph to a Eulerian circuit using only edges that exist in the original graph.

You loop through each edge in the naive Eulerian circuit (naive_euler_circuit). Wherever you encounter an edge that does not exist in the original graph, you replace it with the sequence of edges comprising the shortest path between its nodes using the original graph.

def create_eulerian_circuit(graph_augmented, graph_original, starting_node=None):
    """Create the eulerian path using only edges from the original graph."""
    euler_circuit = []
    naive_circuit = list(nx.eulerian_circuit(graph_augmented, source=starting_node))
    
    for edge in naive_circuit:
        edge_data = graph_augmented.get_edge_data(edge[0], edge[1])    
        
        if edge_data[0]['trail'] != 'augmented':
            # If `edge` exists in original graph, grab the edge attributes and add to eulerian circuit.
            edge_att = graph_original[edge[0]][edge[1]]
            euler_circuit.append((edge[0], edge[1], edge_att)) 
        else: 
            aug_path = nx.shortest_path(graph_original, edge[0], edge[1], weight='distance')
            aug_path_pairs = list(zip(aug_path[:-1], aug_path[1:]))
            
            print('Filling in edges for augmented edge: {}'.format(edge))
            print('Augmenting path: {}'.format(' => '.join(aug_path)))
            print('Augmenting path pairs: {}\n'.format(aug_path_pairs))
            
            # If `edge` does not exist in original graph, find the shortest path between its nodes and 
            #  add the edge attributes for each link in the shortest path.
            for edge_aug in aug_path_pairs:
                edge_aug_att = graph_original[edge_aug[0]][edge_aug[1]]
                euler_circuit.append((edge_aug[0], edge_aug[1], edge_aug_att))
                                      
    return euler_circuit

You hack limitation 3 a bit by starting the Eulerian circuit at the far east end of the park on the Blue trail (node “b_end_east”). When actually running this thing, you could simply skip the last direction which doubles back on it.

Verbose print statements are added to convey what happens when you replace nonexistent edges from the augmented graph with the shortest path using edges that actually exist.

# Create the Eulerian circuit
euler_circuit = create_eulerian_circuit(g_aug, g, 'b_end_east')

Filling in edges for augmented edge: ('y_gy2', 'rs_end_south')
Augmenting path: y_gy2 => y_rs => rs_end_south
Augmenting path pairs: [('y_gy2', 'y_rs'), ('y_rs', 'rs_end_south')]

Filling in edges for augmented edge: ('rc_end_south', 'y_gy1')
Augmenting path: rc_end_south => y_rc => y_gy1
Augmenting path pairs: [('rc_end_south', 'y_rc'), ('y_rc', 'y_gy1')]

Filling in edges for augmented edge: ('v_end_east', 'rs_end_north')
Augmenting path: v_end_east => v_rs => rs_end_north
Augmenting path pairs: [('v_end_east', 'v_rs'), ('v_rs', 'rs_end_north')]

Filling in edges for augmented edge: ('b_bw', 'rh_end_tt_1')
Augmenting path: b_bw => b_tt_1 => rh_end_tt_1
Augmenting path pairs: [('b_bw', 'b_tt_1'), ('b_tt_1', 'rh_end_tt_1')]

Filling in edges for augmented edge: ('rd_end_south', 'v_end_west')
Augmenting path: rd_end_south => b_v => v_end_west
Augmenting path pairs: [('rd_end_south', 'b_v'), ('b_v', 'v_end_west')]

Filling in edges for augmented edge: ('rh_end_north', 'rd_end_north')
Augmenting path: rh_end_north => v_rh => v_rd => rd_end_north
Augmenting path pairs: [('rh_end_north', 'v_rh'), ('v_rh', 'v_rd'), ('v_rd', 'rd_end_north')]

Filling in edges for augmented edge: ('b_tt_3', 'rt_end_north')
Augmenting path: b_tt_3 => b_tt_2 => tt_rt => v_rt => rt_end_north
Augmenting path pairs: [('b_tt_3', 'b_tt_2'), ('b_tt_2', 'tt_rt'), ('tt_rt', 'v_rt'), ('v_rt', 'rt_end_north')]

Filling in edges for augmented edge: ('g_gy1', 'rc_end_north')
Augmenting path: g_gy1 => g_rc => b_rc => v_rc => rc_end_north
Augmenting path pairs: [('g_gy1', 'g_rc'), ('g_rc', 'b_rc'), ('b_rc', 'v_rc'), ('v_rc', 'rc_end_north')]

Filling in edges for augmented edge: ('g_gy2', 'b_end_east')
Augmenting path: g_gy2 => w_gy2 => b_gy2 => b_o => b_y => b_end_east
Augmenting path pairs: [('g_gy2', 'w_gy2'), ('w_gy2', 'b_gy2'), ('b_gy2', 'b_o'), ('b_o', 'b_y'), ('b_y', 'b_end_east')]

You see that the length of the Eulerian circuit is longer than the naive circuit, which makes sense.

print('Length of Eulerian circuit: {}'.format(len(euler_circuit)))

Length of Eulerian circuit: 158

CPP Solution

Text

Here’s a printout of the solution in text:

# Preview first 20 directions of CPP solution
for i, edge in enumerate(euler_circuit[0:20]):
    print(i, edge)

('b_end_east', 'b_y', {'estimate': 0, 'distance': 1.32, 'color': 'blue', 'trail': 'b'})
('b_y', 'y_gy2', {'estimate': 0, 'distance': 0.28, 'color': 'yellow', 'trail': 'y'})
('y_gy2', 'y_rs', {'estimate': 0, 'distance': 0.16, 'color': 'yellow', 'trail': 'y'})
('y_rs', 'rs_end_south', {'estimate': 0, 'distance': 0.39, 'color': 'red', 'trail': 'rs'})
('rs_end_south', 'y_rs', {'estimate': 0, 'distance': 0.39, 'color': 'red', 'trail': 'rs'})
('y_rs', 'y_gy2', {'estimate': 0, 'distance': 0.16, 'color': 'yellow', 'trail': 'y'})
('y_gy2', 'o_gy2', {'estimate': 0, 'distance': 0.12, 'color': 'yellowgreen', 'trail': 'gy2'})
('o_gy2', 'o_rs', {'estimate': 0, 'distance': 0.33, 'color': 'orange', 'trail': 'o'})
('o_rs', 'o_w_2', {'estimate': 0, 'distance': 0.15, 'color': 'orange', 'trail': 'o'})
('o_w_2', 'w_rc', {'estimate': 0, 'distance': 0.23, 'color': 'gray', 'trail': 'w'})
('w_rc', 'y_rc', {'estimate': 0, 'distance': 0.14, 'color': 'red', 'trail': 'rc'})
('y_rc', 'rc_end_south', {'estimate': 0, 'distance': 0.36, 'color': 'red', 'trail': 'rc'})
('rc_end_south', 'y_rc', {'estimate': 0, 'distance': 0.36, 'color': 'red', 'trail': 'rc'})
('y_rc', 'y_gy1', {'estimate': 0, 'distance': 0.18, 'color': 'yellow', 'trail': 'y'})
('y_gy1', 'y_rc', {'estimate': 0, 'distance': 0.18, 'color': 'yellow', 'trail': 'y'})
('y_rc', 'y_rs', {'estimate': 0, 'distance': 0.53, 'color': 'yellow', 'trail': 'y'})
('y_rs', 'o_rs', {'estimate': 0, 'distance': 0.12, 'color': 'red', 'trail': 'rs'})
('o_rs', 'w_rs', {'estimate': 0, 'distance': 0.21, 'color': 'red', 'trail': 'rs'})
('w_rs', 'b_w', {'estimate': 1, 'distance': 0.06, 'color': 'gray', 'trail': 'w'})
('b_w', 'b_gy2', {'estimate': 0, 'distance': 0.41, 'color': 'blue', 'trail': 'b'})

You can tell pretty quickly that the algorithm is not very loyal to any particular trail, jumping from one to the next pretty quickly. An extension of this approach could get fancy and build in some notion of trail loyalty into the objective function to make actually running this route more manageable.

Stats

Let’s peak into your solution to see how reasonable it looks.
(Not important to dwell on this verbose code, just the printed output)

# Computing some stats
total_mileage_of_circuit = sum([edge[2]['distance'] for edge in euler_circuit])
total_mileage_on_orig_trail_map = sum(nx.get_edge_attributes(g, 'distance').values())
_vcn = pd.value_counts(pd.value_counts([(e[0]) for e in euler_circuit]), sort=False)
node_visits = pd.DataFrame({'n_visits': _vcn.index, 'n_nodes': _vcn.values})
_vce = pd.value_counts(pd.value_counts([sorted(e)[0] + sorted(e)[1] for e in nx.MultiDiGraph(euler_circuit).edges()]))
edge_visits = pd.DataFrame({'n_visits': _vce.index, 'n_edges': _vce.values})

# Printing stats
print('Mileage of circuit: {0:.2f}'.format(total_mileage_of_circuit))
print('Mileage on original trail map: {0:.2f}'.format(total_mileage_on_orig_trail_map))
print('Mileage retracing edges: {0:.2f}'.format(total_mileage_of_circuit-total_mileage_on_orig_trail_map))
print('Percent of mileage retraced: {0:.2f}%\n'.format((1-total_mileage_of_circuit/total_mileage_on_orig_trail_map)*-100))

print('Number of edges in circuit: {}'.format(len(euler_circuit)))
print('Number of edges in original graph: {}'.format(len(g.edges())))
print('Number of nodes in original graph: {}\n'.format(len(g.nodes())))

print('Number of edges traversed more than once: {}\n'.format(len(euler_circuit)-len(g.edges())))  

print('Number of times visiting each node:')
print(node_visits.to_string(index=False))

print('\nNumber of times visiting each edge:')
print(edge_visits.to_string(index=False))

Mileage of circuit: 33.59
Mileage on original trail map: 25.76
Mileage retracing edges: 7.83
Percent of mileage retraced: 30.40%

Number of edges in circuit: 158
Number of edges in original graph: 123
Number of nodes in original graph: 77

Number of edges traversed more than once: 35

Number of times visiting each node:
n_nodes  n_visits
     18         1
     38         2
     20         3
      1         4

Number of times visiting each edge:
n_edges  n_visits
     88         1
     35         2

Visualize CPP Solution

While NetworkX also provides functionality to visualize graphs, they are notably humble in this department:

NetworkX provides basic functionality for visualizing graphs, but its main goal is to enable graph analysis rather than perform graph visualization. In the future, graph visualization functionality may be removed from NetworkX or only available as an add-on package.

Proper graph visualization is hard, and we highly recommend that people visualize their graphs with tools dedicated to that task. Notable examples of dedicated and fully-featured graph visualization tools are Cytoscape, Gephi, Graphviz and, for LaTeX typesetting, PGF/TikZ.

That said, the built-in NetworkX drawing functionality with matplotlib is powerful enough for eyeballing and visually exploring basic graphs, so you stick with NetworkX draw for this tutorial.

I used graphviz and the dot graph description language to visualize the solution in my Python package postman_problems. Although it took some legwork to convert the NetworkX graph structure to a dot graph, it does unlock enhanced quality and control over visualizations.

Create CPP Graph

Your first step is to convert the list of edges to walk in the Euler circuit into an edge list with plot-friendly attributes.

create_cpp_edgelist Creates an edge list with some additional attributes that you’ll use for plotting:

sequence: records a sequence of when we walk each edge.
visits: is simply the number of times we walk a particular edge.

def create_cpp_edgelist(euler_circuit):
    """
    Create the edgelist without parallel edge for the visualization
    Combine duplicate edges and keep track of their sequence and # of walks
    Parameters:
        euler_circuit: list[tuple] from create_eulerian_circuit
    """
    cpp_edgelist = {}

    for i, e in enumerate(euler_circuit):
        edge = frozenset([e[0], e[1]])

        if edge in cpp_edgelist:
            cpp_edgelist[edge][2]['sequence'] += ', ' + str(i)
            cpp_edgelist[edge][2]['visits'] += 1

        else:
            cpp_edgelist[edge] = e
            cpp_edgelist[edge][2]['sequence'] = str(i)
            cpp_edgelist[edge][2]['visits'] = 1
        
    return list(cpp_edgelist.values())

Let’s create the CPP edge list:

cpp_edgelist = create_cpp_edgelist(euler_circuit)

As expected, your edge list has the same number of edges as the original graph.

print('Number of edges in CPP edge list: {}'.format(len(cpp_edgelist)))

Number of edges in CPP edge list: 123

The CPP edge list looks similar to euler_circuit, just with a few additional attributes.

# Preview CPP plot-friendly edge list
cpp_edgelist[0:3]

[('o_w_2',
  'w_rs',
  {'color': 'gray',
   'distance': 0.26,
   'estimate': 0,
   'sequence': '79',
   'trail': 'w',
   'visits': 1}),
 ('y_rc',
  'rc_end_south',
  {'color': 'red',
   'distance': 0.36,
   'estimate': 0,
   'sequence': '11, 12',
   'trail': 'rc',
   'visits': 2}),
 ('o_w_1',
  'o_rt',
  {'color': 'orange',
   'distance': 0.13,
   'estimate': 0,
   'sequence': '51, 52',
   'trail': 'o',
   'visits': 2})]

Now let’s make the graph:

# Create CPP solution graph
g_cpp = nx.Graph(cpp_edgelist)

Visualization 1: Retracing Steps

Here you illustrate which edges are walked once (gray) and more than once (blue). This is the "correct" version of the visualization created in 2.4 which showed the naive (as the crow flies) connections between the odd node pairs (red). That is corrected here by tracing the shortest path through edges that actually exist for each pair of odd degree nodes.

If the optimization is any good, these blue lines should represent the least distance possible. Specifically, the minimum distance needed to generate a matching of the odd degree nodes.

plt.figure(figsize=(14, 10))

visit_colors = {1:'lightgray', 2:'blue'}
edge_colors = [visit_colors[e[2]['visits']] for e in g_cpp.edges(data=True)]
node_colors = ['red'  if node in nodes_odd_degree else 'lightgray' for node in g_cpp.nodes()]

nx.draw_networkx(g_cpp, pos=node_positions, node_size=20, node_color=node_colors, edge_color=edge_colors, with_labels=False)
plt.axis('off')
plt.show()

Visualization 2: CPP Solution Sequence

Here you plot the original graph (trail map) annotated with the sequence numbers in which we walk the trails per the CPP solution. Multiple numbers indicate trails we must double back on.

You start on the blue trail in the bottom right (0th and the 157th direction).

plt.figure(figsize=(14, 10))

edge_colors = [e[2]['color'] for e in g_cpp.edges(data=True)]
nx.draw_networkx(g_cpp, pos=node_positions, node_size=10, node_color='black', edge_color=edge_colors, with_labels=False, alpha=0.5)

bbox = {'ec':[1,1,1,0], 'fc':[1,1,1,0]}  # hack to label edges over line (rather than breaking up line)
edge_labels = nx.get_edge_attributes(g_cpp, 'sequence')
nx.draw_networkx_edge_labels(g_cpp, pos=node_positions, edge_labels=edge_labels, bbox=bbox, font_size=6)

plt.axis('off')
plt.show()

Visualization 3: Movie

The movie below that traces the Euler circuit from beginning to end is embedded below. Edges are colored black the first time they are walked and red the second time.

Note that this gif doesn’t do give full visual justice to edges which overlap another or are too small to visualize properly. A more robust visualization library such as graphviz could address this by plotting splines instead of straight lines between nodes.

The code that creates it is presented below as a reference.

First a PNG image is produced for each direction (edge walked) from the CPP solution.

visit_colors = {1:'black', 2:'red'}
edge_cnter = {}
g_i_edge_colors = []
for i, e in enumerate(euler_circuit, start=1):

    edge = frozenset([e[0], e[1]])
    if edge in edge_cnter:
        edge_cnter[edge] += 1
    else:
        edge_cnter[edge] = 1

    # Full graph (faded in background)
    nx.draw_networkx(g_cpp, pos=node_positions, node_size=6, node_color='gray', with_labels=False, alpha=0.07)

    # Edges walked as of iteration i
    euler_circuit_i = copy.deepcopy(euler_circuit[0:i])
    for i in range(len(euler_circuit_i)):
        edge_i = frozenset([euler_circuit_i[i][0], euler_circuit_i[i][1]])
        euler_circuit_i[i][2]['visits_i'] = edge_cnter[edge_i]
    g_i = nx.Graph(euler_circuit_i)
    g_i_edge_colors = [visit_colors[e[2]['visits_i']] for e in g_i.edges(data=True)]

    nx.draw_networkx_nodes(g_i, pos=node_positions, node_size=6, alpha=0.6, node_color='lightgray', with_labels=False, linewidths=0.1)
    nx.draw_networkx_edges(g_i, pos=node_positions, edge_color=g_i_edge_colors, alpha=0.8)

    plt.axis('off')
    plt.savefig('fig/png/img{}.png'.format(i), dpi=120, bbox_inches='tight')
    plt.close()

Then the the PNG images are stitched together to make the nice little gif above.

First the PNGs are sorted in the order from 0 to 157. Then they are stitched together using imageio at 3 frames per second to create the gif.

import glob
import numpy as np
import imageio
import os

def make_circuit_video(image_path, movie_filename, fps=5):
    # sorting filenames in order
    filenames = glob.glob(image_path + 'img*.png')
    filenames_sort_indices = np.argsort([int(os.path.basename(filename).split('.')[0][3:]) for filename in filenames])
    filenames = [filenames[i] for i in filenames_sort_indices]

    # make movie
    with imageio.get_writer(movie_filename, mode='I', fps=fps) as writer:
        for filename in filenames:
            image = imageio.imread(filename)
            writer.append_data(image)

make_circuit_video('fig/png/', 'fig/gif/cpp_route_animation.gif', fps=3)

Next Steps

Congrats, you have finished this tutorial solving the Chinese Postman Problem in Python. You have covered a lot of ground in this tutorial (33.6 miles of trails to be exact). For a deeper dive into network fundamentals, you might be interested in Datacamp’s Network Analysis in Python course which provides a more thorough treatment of the core concepts.

Don’t hesitate to check out the NetworkX documentation for more on how to create, manipulate and traverse these complex networks. The docs are comprehensive with a good number of examples and a series of tutorials.

If you’re interested in solving the CPP on your own graph, I’ve packaged the functionality within this tutorial into the postman_problems Python package on Github. You can also piece together the code blocks from this tutorial with a different edge and node list, but the postman_problems package will probably get you there more quickly and cleanly.

One day I plan to implement the extensions of the CPP (Rural and Windy Postman Problem) here as well. I also have grand ambitions of writing about these extensions and experiences testing the routes out on the trails on my blog here. Another application I plan to explore and write about is incorporating lat/long coordinates to develop (or use) a mechanism to send turn-by-turn directions to my Garmin watch.

And of course one last next step: getting outside and trail running the route!

References

1: Edmonds, Jack (1965). “Paths, trees, and flowers”. Canad. J. Math. 17: 449–467.
2: Galil, Z. (1986). “Efficient algorithms for finding maximum matching in graphs”. ACM Computing Surveys. Vol. 18, No. 1: 23-38.

Intro to graph optimization: solving the Chinese Postman Problem was originally published by andrew brooks at andrew brooks on October 07, 2017.

Exploring convolutional neural networks with DL4J

2016-04-14T00:00:00+00:00

Motivation
Approach
Thoughts

Motivation

TL;DR version: This post walks through an image classification problem hosted on Kaggle for Yelp. I use Scala, DeepLearning4J and convolutional neural networks. For a self-guided tour, check out the project on Github here.

Why…

This project was motivated by a personal desire of mine to:

explore deep learning on a computer vision problem.
implement an end-to-end data science project in Scala.
build an image processing pipeline using real images.

Rather than using the MNIST or CIFAR datasets with pre-processed and standardized images, I wanted to go with a more “wild” dataset of “real-world” images.

I opted for the Kaggle Yelp Restaurant Photo Classification problem. The ~200,000 training images are raw uploads from Yelp users from mobile devices or cameras with a variety of sizes, dimensions, colors and quality.

What I did instead…

I was initially going to document this project end-to-end from image processing to training the convolutional neural networks. However, upon more research and practice actually tuning convolutional networks, I’ve reconsidered my process. While the Kaggle Yelp Photo Classification problem is a novel problem, it turns out not to be a great match with the deep learning techniques I wanted to explore. Thus, this article will focus mainly on the image processing pipeline using Scala. While I introduce DL4J here, I plan to discuss my experience with it in more detail in a forthcoming post.

The Kaggle problem

The Kaggle problem is this. Yelp wants to auto-classify restaurants on the 9 characteristics below:

good_for_lunch
good_for_dinner
takes_reservations
outdoor_seating
restaurant_is_expensive
has_alcohol
has_table_service
ambience_is_classy
good_for_kids

Each restaurant has some number of images (from a couple to several hundred). However there are no restaurant features beyond these images. Thus it is a multiple-instance learning problem where each business in the training data is represented by its bag of images.

This is also a multiple-label classification problem where each business can have one or more of the 9 characteristics listed above.

Initial approach

To deal with the multiple-instance issue, I simply applied the labels of the restaurant to all of the images associated with it and treated each image as a separate record.

To deal with with the multiple-label problem, I simply handled each class as a separate binary classification problem. While there are breeds of neural networks capable of classifying multiple labels, such as BP-MLL, these are not currently available in DL4J.

Pivot

While I didn’t expect my initial approach would land me at the top of the Kaggle leaderboard, I did expect it would allow me to build a reasonable model while exploring new and untested (to me) tools and techniques: DeepLearning4j, Scala and convolutional nets. That assumption turned out to bigger than I expected.

The noise-to-signal ratio turned out to be too high with the Yelp data to train a meaningful convolutional network given my self-imposed constraints. From what I’ve deduced from the Kaggle forum, most teams are using pre-trained neural networks to extract features from each image. From there it can be tackled as a classical (non-image) classification problem with crafty feature creation and aggregation from the image to restaurant level.

While this is far more computationally efficient and could yield better predictions, it cuts out exactly the part I wanted to explore. I eventually compromised with myself and decided to re-factor the image pipeline I developed on this project for a similar better posed problem using CIFAR or dataset created myself from using image-net.

Approach

Image processing

Images in the training set come in various shapes and sizes. See some examples below. My first pass at processing consists of:

squaring images
resizing image to same same dimensions
grayscaling image

Some images are tall…

Some images are wide…

Some images are outside…

Some images are inside…

Some images are food…

And some are random other things…

1. Square images

While images in the training set varied from portrait to landscape and the number of pixels, most were roughly square. Many were exactly 500 x 375, which was also the largest size, presumably the output of Yelp’s own image processing system.

To train a convolutional net, all images need to be the same shape and size. While there are likely fancier tricks and techniques that allow for different sized images, I started simple: make all images square, while preserving as much of the image as possible. I assume that the material of interest is centered, so I capture the middle-most square of each image.

Example:

original 500 x 375

squared 375 x 375

This example was created with the following code:

2. Re-size images

Now that images are squared, the re-sizing problem is relatively straightforward.

Example:

original 500 x 375

re-sized 256 x 256

re-sized 128 x 128

re-sized 64 x 64

re-sized 32 x 32

This example was created with the following code:

3. Grayscale

While DL4J and convolutional nets can certainly handle color images, I decided to simplify computation and start with grayscale. This way a single 64 x 64 pixel image is represented by 4096 features rather than 4096*3 (one for each color channel: R, G, B). There is a good discussion of the numerous ways to do this here. I opted to start with the simplest of all (averaging) which appeared to work quite well. Here’s an example:

original image (left); grayscale conversion using RGB averaging (right)

This example was created with the following code:

Pipeline - images

Much of this section is specific to the Kaggle problem and discusses the data structures I created and used to keep store and manage images with their corresponding labels. It’s mainly an exploration of how to structure a data science project with Scala. If you’re primarily interested in DL4J, skip ahead to the Pipeline - DL4J section.

Image processing

In my image processing pipeline, I modified the functions in the Gists above to methods of the java.awt.image.BufferedImage class.

This allows me to operate on images with chaining like this:

import imgUtils._

 val img = ImageIO.read(new File("myimagefile.jpg"))
         .makeSquare
         .resizeImg(128, 128)
         .image2gray

I’m not sure if this approach of extending an existing class with new methods is preferred to creating a new class, but it seemed to work well for my problem. I imagine it would be less clean if all instances of the original class do not need the newly defined methods. However, this wasn’t the case for me: all images need the new methods.

Code below: extendBufferedImage.scala

package modeling.processing

import scala.Vector
import org.imgscalr._

object imgUtils {
   implicit class extendingImageClass(img: java.awt.image.BufferedImage) {
    
    // image 2 vector processing
    def pixels2gray(red: Int, green:Int, blue: Int): Int = (red + green + blue) / 3
    def pixels2color(red: Int, green:Int, blue: Int): Vector[Int] = Vector(red, green, blue)
  
    private def image2vec[A](f: (Int, Int, Int) => A ): Vector[A] = {
               val w = img.getWidth
               val h = img.getHeight
               for { w1 <- (0 until w).toVector
                     h1 <- (0 until h).toVector
                   } yield {
                       val col = img.getRGB(w1, h1)
        			         val red =  (col & 0xff0000) / 65536
        			         val green = (col & 0xff00) / 256
        			         val blue = (col & 0xff)
                       f(red, green, blue)
                   }
             }
    
    def image2gray: Vector[Int] = image2vec(pixels2gray)
    def image2color: Vector[Int] = image2vec(pixels2color).flatten
    
    // make image square
    def makeSquare = {
      val w = img.getWidth
      val h = img.getHeight
      val dim = List(w, h).min
      
      img match {
    	  case x if w == h => img
    	  case x if w > h => Scalr.crop(img, (w-h)/2, 0, dim, dim)
    	  case x if w < h => Scalr.crop(img, 0, (h-w)/2, dim, dim)
      }
    }                        
    
    // resize pixels
    def resizeImg(width: Int, height: Int) = {
      Scalr.resize(img, Scalr.Method.BALANCED, width, height)
    }
    
   }
}

I/O

We need to load a couple CSV files containing metadata about each image. There are some Scala CSV reader libraries out there like scala-csv, however I forwent these to get more experience testing out Scala. I defined a basic file-reader readcsv which is used by readBizLabels and readBiz2ImgLabels to read in text files containing the labels for each Yelp business and image-to-business mappings respectively.

Code below: readCsvData.scala

package modeling.io

import scala.io.Source

object readCsvData {
  
  /** Generic function to load in CSV */
  
  def readcsv(csv: String, rows: List[Int]=List(-1)):  List[List[String]] = {
    val src = Source.fromFile(csv) 
    def reading(csv: String): List[List[String]] = {  
      src.getLines.map(x => x.split(",").toList)
         .toList
    }
    try {
        if(rows==List(-1)) reading(csv)
        else rows.map(reading(csv))
    } finally {
        src.close
    }
  }
  
  /** Create map from bizid to labels of form bizid -> Set(labels)  */
  
  def readBizLabels(csv: String, rows: List[Int]=List(-1)): Map[String, Set[Int]]  = {
    val src = readcsv(csv)
    src.drop(1) // drop header
       .map(x => x match {
          case x :: Nil => (x(0).toString, Set[Int]())
          case _ => (x(0).toString, x(1).split(" ").map(y => y.toInt).toSet)
          }).toMap
  }
  
  /** Create map from imgID to bizID of form imgID -> busID  */
   
  def readBiz2ImgLabels(csv: String, rows: List[Int] = List(-1)): Map[Int, String]  = {
    val src = readcsv(csv)
    src.drop(1) // drop header
       .map(x => x match {
         case x :: Nil => (x(0).toInt, "-1")
          case _ => (x(0).toInt, x(1).split(" ").head)
       }).toMap
  }

Wrangling

I make heavy use of the Scala map class. Essentially we have three maps:

bizMap (imgID -> bizID)
dataMap (imgID -> img data)
labMap (bizID -> labels)

I suppose I could have made classes for each of these as well, but they’re really just intermediate data structures, so I didn’t bother.

readBizLabels from the code above creates the bizMap and readBiz2ImgLabels creates the imgMap. processImages from the code below creates the dataMap. Next step: create a single data representation of these three separate but related data structures.

Code below: images.scala

package modeling.processing

import java.io.File
import javax.imageio.ImageIO
import scala.util.matching.Regex
import imgUtils._

object images {
  
  /**  Define RegEx to extract jpg name from the image class which is used to match against training labels */
  val patt_get_jpg_name = new Regex("[0-9]")
  
  /** Collects all images associated with a BizId. */
  def getImgIdsForBizId(bizMap: Map[Int, String], bizIds: List[String]): List[Int] = {
    bizMap.filter(x => bizIds.exists(y => y == x._2)).map(_._1).toList
  }
  
   /** Get a list of images to load and process
      *
      * @param photoDir directory where the raw images reside
      * @param ids optional parameter to subset the images loaded from photoDir.
      * 
      * @example println(getImageIds("/Users/abrooks/Documents/kaggle_yelp_photo/train_photos/", ids=List.range(0,10)))
      */
  
  def getImageIds(photoDir: String, bizMap: Map[Int, String] = Map(-1 -> "-1"), bizIds: List[String] = List("-1")): List[String] = {
    val d =  new File(photoDir) // new File("data/images/") // too many photos?
    val imgsPath = d.listFiles().map(x => x.toString).toList
    
    if (bizMap == Map(-1 -> "-1") || bizIds == List(-1)) {
      imgsPath 
    } else {
      val imgsMap = imgsPath.map(x => patt_get_jpg_name.findAllIn(x).mkString.toInt -> x).toMap
      val imgsPathSub = getImgIdsForBizId(bizMap, bizIds)
      imgsPathSub.map(x => imgsMap(x))
    }
  }
  
   /** Read and process images into a photoID -> vector map
      *
      * @param imgs list of images to read-in.  created from getImageIds function.
      * @param resizeImgDim dimension to rescale square images to
      * @param nPixels number of pixels to maintain.  mainly used to sample image to drastically reduce runtime while testing features. 
      * 
      * @example
        val imgs = getImageIds("/Users/abrooks/Documents/kaggle_yelp_photo/train_photos/", ids=List(0,1,2,3,4))
        println(processImages(imgs, resizeImgDim = 128, nPixels = 16))    */
  
  def processImages(imgs: List[String], resizeImgDim: Int = 128, nPixels: Int = -1): Map[Int, Vector[Int]] = {       
    imgs.map(x => 
      patt_get_jpg_name.findAllIn(x).mkString.toInt -> { 
        val img0 = ImageIO.read(new File(x))
         .makeSquare
         .resizeImg(resizeImgDim, resizeImgDim) // (200, 200)
         .image2gray
       if(nPixels != -1) img0.slice(0, nPixels)
       else img0
     }   
   ).filter( x => x._2 != ())
    .toMap
    
  }

Data structure

So there are four pieces of information to keep track of for each image:

imageID
bizID
labels
pixel data

The data is represented like this:

I defined a class alignedData to manage it all. When instantiating an instance of alignedData, the bizMap, dataMap and labMap are provided. I used Scala’s Option type for labMap since we don’t have this information when we score test data. None is provided that case.

Under the hood, the primary data structure has the following type:

List[(Int, String, Vector[Int], Set[Int])]

which corresponds to a list of Tuple4s containing this information:

List[(imgID, bizID, pixel data vector, labels)]

Code below: alignedData.scala

package modeling.processing

class alignedData(dataMap: Map[Int, Vector[Int]], bizMap: Map[Int, String], labMap: Option[Map[String, Set[Int]]])
                 (rowindices: List[Int] = dataMap.keySet.toList) {
  
  // initializing alignedData with empty labMap when it is not provided (we are working with training data)
  def this(dataMap: Map[Int, Vector[Int]], bizMap: Map[Int, String])(rowindices: List[Int]) = this(dataMap, bizMap, None)(rowindices)

  def alignBizImgIds(dataMap: Map[Int, Vector[Int]], bizMap: Map[Int, String])
    (rowindices: List[Int] = dataMap.keySet.toList): List[(Int, String, Vector[Int])] = {
      for { pid <- rowindices
          val imgHasBiz = bizMap.get(pid) // returns None if img does not have a bizID
          val bid = if(imgHasBiz != None) imgHasBiz.get else "-1"
          if (dataMap.keys.toSet.contains(pid) && imgHasBiz != None)
      } yield { 
          (pid, bid, dataMap(pid))
      }
  }
  
  def alignLabels(dataMap: Map[Int, Vector[Int]], bizMap: Map[Int, String], labMap: Option[Map[String, Set[Int]]])
    (rowindices: List[Int] = dataMap.keySet.toList): List[(Int, String, Vector[Int], Set[Int])] = {
      def flatten1[A, B, C, D](t: ((A, B, C), D)): (A, B, C, D) = (t._1._1, t._1._2, t._1._3, t._2)
      val al = alignBizImgIds(dataMap, bizMap)(rowindices)
      for { p <- al
      } yield {
        val bid = p._2
        val labs = labMap match  {
          case None => Set[Int]()
          case x => (if(x.get.keySet.contains(bid)) x.get(bid) else Set[Int]())
        }
        flatten1(p, labs) 
      }
  }
  
  // pre-computing and saving data as a val so method does not need to re-compute each time it is called. 
  lazy val data = alignLabels(dataMap, bizMap, labMap)(rowindices)
  
  // getter functions
  def getImgIds = data.map(_._1)
  def getBizIds = data.map(_._2)
  def getImgVectors = data.map(_._3)
  def getBizLabels = data.map(_._4) 
  def getImgCntsPerBiz = getBizIds.groupBy(identity).mapValues(x => x.size) 
  
}

Make ND4J dataset

Last step is to create the data structure that DL4J needs for training convolutional nets. That data structure is an ND4J DataSet.

This is relatively straightforward once you figure out how to convert native Scala data structures to this type.

Code below: makeDataSets.scala

package modeling.processing

import org.nd4j.linalg.dataset.{DataSet}
import org.nd4s.Implicits._ 
import org.nd4j.linalg.api.ndarray.INDArray

import alignedData._

object makeDataSets {
  
  /** Creates DataSet object from the data structure of data structure from alignLables function in form List[(imgID, bizID, labels, pixelVector)]  */
  
  def makeDataSet(alignedData: alignedData, bizClass: Int): DataSet = {
    val alignedXData = alignedData.getImgVectors.toNDArray
    val alignedLabs = alignedData.getBizLabels.map(x => if (x.contains(bizClass)) Vector(1, 0) else Vector(0, 1)).toNDArray
    new DataSet(alignedXData, alignedLabs) 
  }
  
  def makeDataSetTE(alignedData: alignedData): INDArray = {
    alignedData.getImgVectors.toNDArray
  }
  
}

Run

The code to actually run the image processing pipeline boils down to the following:

Code below: snippet from main.scala

import io.readCsvData._

import processing.alignedData
import processing.images._
import processing.makeDataSets._

object runPipeline {
  def main(args: Array[String]): Unit = {
    
  // image processing on training data
  val labMap = readBizLabels("data/labels/train.csv")
  val bizMap = readBiz2ImgLabels("data/labels/train_photo_to_biz_ids.csv")
  val imgs = getImageIds("data/images/train", bizMap, bizMap.map(_._2).toSet.toList)
  val dataMap = processImages(imgs, resizeImgDim = 64)
  val alignedData = new alignedData(dataMap, bizMap, Option(labMap))()

Pipeline - DL4J

I didn’t find many examples of DL4J applications in Scala… one of the reasons I’m documenting this project in detail. However, there are some useful examples here.

Batch-mode

This took some exploring to figure out. The default speficification for DL4J networks that run with the ND4J DataSet do not train in batches. That is, each epoch (full pass through the training examples) will train on all training examples in one computation step. So all images, their data and corresponding weights must be held in memory at once.

This hogs memory with all but the smallest of datasets. I was running into heap space errors with just ~2,000 images sized 128 x 128. After switching to batch-mode, I was able to train on tens of thousands of images without memory issues.

Before I discovered this was the cause of my heap space problem, I posed my problem to the project contributors on the DL4J Gitter. I was pleased to learned that the next release of DL4J (3.9) is planned to move some computational operations off heap.

How to use batches

It’s easier to figure this out now that the DL4J examples repo has a convolutional net example (MNIST) using batches, which as of a few weeks ago was not there.

The biggest difference from mini-batch to full-batch mode is that you need to pass a MultipleEpochsIterator object rather than a ND4J DataSet to the fit method of your MultiLayerNetwork object. My approach doesn’t fully embrace iterators for their intended purpose, but hey it works and made for a smooth transition using my pipeline. You also need to add .miniBatch(true) to your MultiLayerConfiguration.Builder.

The distinction between iterations and epochs can be slightly confusing when moving from full-batch to mini-batch mode. If you’re not using miniBatches, the iterations method is used to specify how many epochs you want. However, when using batches, this is done directly in MultipleEpochsIterator and iterations can be set to 1. Explained here in the DL4J documentation.

import org.deeplearning4j.datasets.iterator.MultipleEpochsIterator
import org.deeplearning4j.datasets.iterator.impl.ListDataSetIterator
 
val nbatches = 128
val nepochs = 100
val dsiterTr = new ListDataSetIterator(trainTest.getTrain.asList(), nbatch)
val dsiterTe = new ListDataSetIterator(trainTest.getTest.asList(), nbatch)
val epochitTr: MultipleEpochsIterator = new MultipleEpochsIterator(nepochs, dsiterTr)
val epochitTe: MultipleEpochsIterator = new MultipleEpochsIterator(nepochs, dsiterTe)

Train convolutional network

This is the function that actually trains the convolutional net. It’s is long, probably too long. Lots of hyper-parameters hardcoded within that could be moved to function arguments or better yet, a config file. However, I was running locally on laptop with lots of tinkering, so this worked fine for me.

The CNN training function below does a lot: logging, test/train splitting, creating the MultipleEpochsIterator, training, reporting performance on test data, and saving the trained models to file.

I’ll save the intuition behind tuning for another post after I gain a better understanding myself on a more well posed problem. For now I’ll just ramble about what I tried and what happened.

Layers: I observed from some papers benchmarking convolutional nets solving the MNIST problem that a single convolutional layer generates decent results (certainly better than a benchmark of random, at least enough get started with). I trained with runs with up to three convolutional layers without errors, but my training was obviously slower (although not exponentially) and results were not any better than with one layer. Next time I plan to start with one convolutional layer, start tuning other parameters to get above benchmark results… and then explore additional layers.

# of samples: Training on all images took so much time, I didn’t have the patience to let it finish. It took me about 2 days to run a watered down CNN on my laptop with 50,000 images.

nepochs: This is the number of passes through all the training records. I’ve seen some networks with as few as 20 to as many as 1000 epochs. This is the main idea. There’s also an in-depth discussion about this here. I spent most of my tuning time trading off nepochs and the # of samples. I could tolerate training with lots of images to expose the network to a broader universe of features to learn… but only by cutting down the number of epochs to make run-time manageable.

nOut: This is the number feature maps. I tried runs with 10 to 500, chosen mostly by reviewing configurations for other image problems and the example DL4J networks.

learningRate: From what I’ve read this is pretty important. I didn’t fiddle with this much though. I think I tried the commonly used .01 and .001.

nbatch: This is the # of records in each batch. I tried 32, 64 and 128. I’m not sure how much of a difference this makes for results vs. computation.

Code below: cnnEpochs.scala. A simpler version using full-batch training is here.

package modeling.training

import modeling.processing.makeDataSets._
import modeling.io._
import modeling.processing.alignedData
import java.util.Random
import java.nio.file._
import java.io.{DataOutputStream, File}
import org.apache.commons.io.FileUtils
import org.deeplearning4j.eval.Evaluation
import org.deeplearning4j.nn.api.OptimizationAlgorithm
import org.deeplearning4j.nn.conf.layers.setup.ConvolutionLayerSetup
import org.deeplearning4j.nn.conf.layers.{ConvolutionLayer, OutputLayer, SubsamplingLayer, DenseLayer}
import org.deeplearning4j.nn.conf.{MultiLayerConfiguration, NeuralNetConfiguration}
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork
import org.deeplearning4j.nn.weights.WeightInit
import org.deeplearning4j.optimize.api.IterationListener
import org.deeplearning4j.optimize.listeners.ScoreIterationListener
import org.nd4j.linalg.api.ndarray.INDArray
import org.nd4j.linalg.dataset.SplitTestAndTrain
import org.nd4j.linalg.factory.Nd4j
import org.nd4j.linalg.lossfunctions.LossFunctions
import org.slf4j.LoggerFactory
import scala.collection.JavaConverters._
import org.deeplearning4j.datasets.iterator.MultipleEpochsIterator
import org.deeplearning4j.datasets.iterator.impl.ListDataSetIterator

object cnnEpochs {
  
  def trainModelEpochs(alignedData: alignedData, bizClass: Int = 1, saveNN: String = "") = {
    
    val ds = makeDataSet(alignedData, bizClass)

    println("commence training!!")
    println("class for training: " + bizClass)

    val begintime = System.currentTimeMillis()
  
    lazy val log = LoggerFactory.getLogger(cnn.getClass)
    log.info("Begin time: " + java.util.Calendar.getInstance().getTime())
  
      val nfeatures = ds.getFeatures.getRow(0).length // hyper, hyper parameter
      
      val numRows =  Math.sqrt(nfeatures).toInt // numRows * numColumns must equal columns in initial data * channels
      val numColumns = Math.sqrt(nfeatures).toInt // numRows * numColumns must equal columns in initial data * channels
      val nChannels = 1 // would be 3 if color image w R,G,B
      val outputNum = 2 // # of classes (# of columns in output)
      val iterations = 1
      val splitTrainNum = math.ceil(ds.numExamples*0.8).toInt // 80/20 training/test split
      val seed = 123
      val listenerFreq = 1
      val nepochs = 20
      val nbatch = 128 // recommended between 16 and 128
      
      //val nOutPar = 500 // default was 1000.  # of output nodes in first layer
  
      println("rows: " + ds.getFeatures.size(0))
      println("columns: " + ds.getFeatures.size(1))
      
      /**
       *Set a neural network configuration with multiple layers
       */
      log.info("Load data....")
      ds.normalizeZeroMeanZeroUnitVariance() // this changes ds
      System.out.println("Loaded " + ds.labelCounts)
      Nd4j.shuffle(ds.getFeatureMatrix, new Random(seed), 1) // this changes ds.  Shuffles rows
      Nd4j.shuffle(ds.getLabels, new Random(seed), 1) // this changes ds.  Shuffles labels accordingly
      val trainTest: SplitTestAndTrain = ds.splitTestAndTrain(splitTrainNum, new Random(seed)) // Random Seed not needed here
      
      
      // creating epoch dataset iterator
      val dsiterTr = new ListDataSetIterator(trainTest.getTrain.asList(), nbatch)
      val dsiterTe = new ListDataSetIterator(trainTest.getTest.asList(), nbatch)
      val epochitTr: MultipleEpochsIterator = new MultipleEpochsIterator(nepochs, dsiterTr)
      val epochitTe: MultipleEpochsIterator = new MultipleEpochsIterator(nepochs, dsiterTe)
    
      val builder: MultiLayerConfiguration.Builder = new NeuralNetConfiguration.Builder()
              .seed(seed)
              .iterations(iterations)
              .miniBatch(true)
              .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
              .learningRate(0.01)
              .momentum(0.9)
              .list(4)
              .layer(0, new ConvolutionLayer.Builder(6,6)
                      .nIn(nChannels)
                      .stride(2,2) // default stride(2,2)
                      .nOut(20) // # of feature maps
                      .dropOut(0.5)
                      .activation("relu") // rectified linear units
                      .weightInit(WeightInit.RELU)
                      .build())
              .layer(1, new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX, Array(2,2))
                      .build())
              .layer(2, new DenseLayer.Builder()
                      .nOut(40)
                      .activation("relu")
                      .build())
              .layer(3, new OutputLayer.Builder(LossFunctions.LossFunction.MCXENT)
                      .nOut(outputNum)
                      .weightInit(WeightInit.XAVIER)
                      .activation("softmax")
                      .build())
              .backprop(true).pretrain(false)
              
      new ConvolutionLayerSetup(builder, numRows, numColumns, nChannels)
              
      val conf: MultiLayerConfiguration = builder.build()

      log.info("Build model....")
      val model: MultiLayerNetwork = new MultiLayerNetwork(conf)
      model.init() 
      model.setListeners(Seq[IterationListener](new ScoreIterationListener(listenerFreq)).asJava)

      log.info("Train model....")
      System.out.println("Training on " + dsiterTr.getLabels) // this might return null
      model.fit(epochitTr)
      
      // TRAINING
      log.info("Evaluate model....")
      System.out.println("Testing on ...")
      val eval = new Evaluation(outputNum)
        while(epochitTe.hasNext) {
            val testDS = epochitTe.next(nbatch)
            val output: INDArray = model.output(testDS.getFeatureMatrix)
            eval.eval(testDS.getLabels(), output)
        }
        System.out.println(eval.stats())
      
      
      val endtime = System.currentTimeMillis()
      log.info("End time: " + java.util.Calendar.getInstance().getTime())
      log.info("computation time: " + (endtime-begintime)/1000.0 + " seconds")
      
      log.info("Write results....")
      
      if(!saveNN.isEmpty) { 
        // model config
        FileUtils.write(new File(saveNN + ".json"), model.getLayerWiseConfigurations().toJson()) 
        
        // model parameters
        val dos: DataOutputStream = new DataOutputStream(Files.newOutputStream(Paths.get(saveNN + ".bin")))
        Nd4j.write(model.params(), dos)
      }
    
      log.info("****************Example finished********************")
          
  }

  
}

Save/load networks

This is definitely something you’ll want to do. These things take too long to train to not save immediately.

saveNN is pretty straightforward. It saves a .json file with the network configuration and a .bin with all the weights and parameters of the network you just trained.

loadNN just reads back the .json and .bin file you created with saveNN to a MultiLayerNetwork object that you can use to score new test data.

Code below: nn.scala

package modeling.io

import org.deeplearning4j.nn.conf.{GradientNormalization, MultiLayerConfiguration, NeuralNetConfiguration}
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork

import org.nd4j.linalg.factory.Nd4j

import java.io.File
import org.apache.commons.io.FileUtils
import java.io.{DataInputStream, DataOutputStream, FileInputStream}
import java.nio.file.{Files, Paths}


object nn {
  
  def loadNN(NNconfig: String, NNparams: String) = {
    // get neural network config
    val confFromJson: MultiLayerConfiguration = MultiLayerConfiguration.fromJson(FileUtils.readFileToString(new File(NNconfig)))
    
     // get neural network parameters 
    val dis: DataInputStream = new DataInputStream(new FileInputStream(NNparams))
    val newParams = Nd4j.read(dis)
    
     // creating network object
    val savedNetwork: MultiLayerNetwork = new MultiLayerNetwork(confFromJson)
    savedNetwork.init()
    savedNetwork.setParameters(newParams)
    
    savedNetwork
  }
  
  def saveNN(model: MultiLayerNetwork, NNconfig: String, NNparams: String) = {
    // save neural network config
    FileUtils.write(new File(NNconfig), model.getLayerWiseConfigurations().toJson()) 
    
    // save neural network parms
    val dos: DataOutputStream = new DataOutputStream(Files.newOutputStream(Paths.get(NNparams)))
    Nd4j.write(model.params(), dos)
  }
  
}

Scoring

I won’t say much here, since I didn’t end up putting much emphasis on this step for reasons explained at the beginning of this post.

My scoring approach assigns business-level labels by averaging the image-level predictions. I classify a business as label “0” if the average of the probabilities across all of its images belonging class “0” is greater than 0.5.

Code below: scoring.scala

package modeling.processing

import org.nd4j.linalg.api.ndarray.INDArray
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork

object scoring {
  
  def scoreModel(model: MultiLayerNetwork, ds: INDArray) = {
    model.output(ds)
  }
  
  /** Take model predictions from scoreModel and merge with alignedData*/
  
  def aggImgScores2Biz(scores: INDArray, alignedData: alignedData ) = {
    assert(scores.size(0) == alignedData.data.length, "alignedData and scores length are different.  They must be equal")
    def getRowIndices4Biz(mylist: List[String], mybiz: String): List[Int] = mylist.zipWithIndex.filter(x => x._1 == mybiz).map(_._2)
    def mean(xs: List[Double]) = xs.sum / xs.size

    alignedData.getBizIds.distinct.map(x => (x, {
      val irows = getRowIndices4Biz(alignedData.getBizIds, x)
      val ret = for(row <- irows) yield scores.getRow(row).getColumn(1).toString.toDouble
      mean(ret)
    }))
    
  }
  
}

Submit to Kaggle

Also not much to say here, but this is how I aggregated image predictions to business scores for each model. And this is the code to generate the output CSV for Kaggle.

Run

The whole project can be run from main.scala. Here it is:

package modeling

import io.nn._
import io.kaggleSubmission._ 
import io.readCsvData._

import processing.alignedData
import processing.images._
import processing.kaggleSubmission._
import processing.makeDataSets._
import processing.scoring._

import training.cnn._
import training.cnnEpochs._

object runPipeline {
  def main(args: Array[String]): Unit = {
    
  // image processing on training data
  val labMap = readBizLabels("data/labels/train.csv")
  val bizMap = readBiz2ImgLabels("data/labels/train_photo_to_biz_ids.csv")
  val imgs = getImageIds("data/images/train", bizMap, bizMap.map(_._2).toSet.toList).slice(0,20000) // 20000 images
  val dataMap = processImages(imgs, resizeImgDim = 128) 
  val alignedData = new alignedData(dataMap, bizMap, Option(labMap))()
  
  // training (one model/class at a time). Many microparameters hardcoded within
  val cnn0 = trainModelEpochs(alignedData, bizClass = 0, saveNN = "results/modelsV0/model0") 
  val cnn1 = trainModelEpochs(alignedData, bizClass = 1, saveNN = "results/modelsV0/model1") 
  val cnn2 = trainModelEpochs(alignedData, bizClass = 2, saveNN = "results/modelsV0/model2")
  val cnn3 = trainModelEpochs(alignedData, bizClass = 3, saveNN = "results/modelsV0/model3")
  val cnn4 = trainModelEpochs(alignedData, bizClass = 4, saveNN = "results/modelsV0/model4")
  val cnn5 = trainModelEpochs(alignedData, bizClass = 5, saveNN = "results/modelsV0/model5")
  val cnn6 = trainModelEpochs(alignedData, bizClass = 6, saveNN = "results/modelsV0/model6")
  val cnn7 = trainModelEpochs(alignedData, bizClass = 7, saveNN = "results/modelsV0/model7")
  val cnn8 = trainModelEpochs(alignedData, bizClass = 8, saveNN = "results/modelsV0/model8")

  // processing test data for scoring
  val bizMapTE = readBiz2ImgLabels("data/labels/test_photo_to_biz.csv")
  val imgsTE = getImageIds("data/images/test/", bizMapTE, bizMapTE.map(_._2).toSet.toList)
  val dataMapTE = processImages(imgsTE, resizeImgDim = 128)
  val alignedDataTE = new alignedData(dataMapTE, bizMapTE, None)()
  
  // creating csv file to submit to kaggle (scores all models)
  val kaggleResults = createKaggleSubmitObj(alignedDataTE, "results/ModelsV0/")
  val kaggleSubmitResults = writeKaggleSubmissionFile("results/kaggleSubmission/kaggleSubmitFile.csv", kaggleResults, thresh = 0.5)

Thoughts

This was my first foray into deep neural networks. I haven’t used theano or any of the other widely used implementations out there, so I unfortunately don’t have much to compare my experience to.

I will say that the current documentation will only take you so far. I spent a lot of time reading Neural Networks and Deep Learning to understand the concepts and reviewing the DL4J source code to try and figure out how to implement what I thought I wanted to do.

Discovering the DL4J Gitter was the single most useful moment I had. The creators are actively answering all sorts of questions in real-time. There’s also a room for earlyadopters discussing testing and feature requests which was interesting to browse. Very impressed with the commitment and willingness to help. I even got an email from someone on the DL4J team after I pushed this project to GitHub offering to help and pointing me to the CNN specialists.

Gitter is where the action is. There’s way more here than on StackOverflow. However, the content doesn’t appear to be indexed nearly as well on Google, so found myself “Googling” in the Gitter search bar for keywords and perusing through conversations to get answers.

I recommend using the deeplearning4j-ui tool if you can. I unfortunately wasn’t able to get it working, but it looks super useful for understanding how your net training is going.

Other awesome resources I found for visualizing training for CNNs are ConvNetJS and this one.

Exploring convolutional neural networks with DL4J was originally published by andrew brooks at andrew brooks on April 14, 2016.

Managing managed libraries with Scala and Eclipse

2015-12-29T00:00:00+00:00

My story [skip for the real answer]

I have become increasingly intrigued with the functional programming paradigm and its implementation in Scala. I’m thoroughly enjoying working my way through Scala creator Martin Odersky’s coursera course which I highly recommend to anyone from “timid and interested” to “hardcore.” As a data scientist with a statistics and economics background, I code every day in languages like R and Python wrangling data, implementing machine learning algorithms, tapping APIs and building data tools for data scientists. However, I don’t build [professional-grade] software. Ive never taken a formal computer science course in college/grad school or learned Java. Admittedly, I’ve felt overwhelmed at times by so many new concepts and tools with the course. I’m simultaneously figuring out Scala, Java, sbt, eclipse, functional and object oriented paradigms.

As a learner poking my head in the field, I find it refreshing reading stories from fellow newcomers battling similar problems, even if they appear trivial in hindsight. I also find it most effective teaching material when you still remember how you learned it. Thus, this post and [likely, yet currently unwritten] subsequent Scala posts will be experimental learning notes rather than wisdom and tested best practices.

Managed vs Unmanaged dependencies

One of the stumbling blocks I encountered on my first Scala project was a simple one: working with external libraries with sbt and Eclipse. I started using unmanaged dependencies (downloading some jar files and pointing Eclipse to them) which worked well, until I encountered a library which had further dependencies – I was working with json4s. More on unmanaged vs managed dependencies here. Managed dependencies have the advantage of being, well, managed. You simply specify a library with a version and Ivy (a tool similar to Maven) will go out on the web and download it along with any dependencies.

My issue was I initially couldn’t get Eclipse to actually go out and download my dependencies. As evidenced by this StackOverflow post, there are probably many ways to achieve this. The piece I was missing was using sbt in the Terminal while Eclipse was running to actually download the libraries.

Make Eclipse recognize managed dependencies

Here’s a step-by-step process

Step 1: Create a build.sbt file

Create build.sbt in the root project directory. For example, my build.sbt looks like this:

name := "billscraper"

scalaVersion := "2.11.5"

libraryDependencies ++= Seq( 
  "org.json4s" %% "json4s-native" % "3.2.11",
  "org.json4s" %% "json4s" % "3.2.11",
  "io.spray" %% "spray-json" % "1.3.2",
  "org.scalanlp" %% "breeze" % "0.11.2"
)

Step 2: Open the Terminal

Assuming you’re on mac, type the following in the Terminal:

> sbt
> eclipse

You should see a bunch of jar files downloading… something like this:

Step 3: Refresh project in Eclipse

Now in Eclipse refresh your project by clicking F5, or right click on the project in the Package Explorer tab and click “Refresh”

Step 4: Check that it worked

Now to check that libraries have downloaded and Eclipse has recognized them.

In Eclipse, right click on the project in the Package Explorer tab
   => click “Build Path”
   => click “Configure Build Path”
   => click “Libraries” (icon with a stack of books)

Here you should see the libraries from build.sbt and all their dependencies. Now you should be good to go – import what you need from these libraries in your Scala code.

My environment

I configured my environment as recommended by the Functional Programming Principles in Scala course:

Mac - OSX Yosemite 10.10
Scala IDE build of Eclipse SDK 4.2.0
sbt 0.13.9

Managing managed libraries with Scala and Eclipse was originally published by andrew brooks at andrew brooks on December 29, 2015.

Interactive association rules exploration app

2015-11-30T00:00:00+00:00

Features
How to get
How to use
Screenshots
Code

In a previous post, I wrote about what I use association rules for and mentioned a Shiny application I developed to explore and visualize rules. This post is about that app. The app is mainly a wrapper around the arules and arulesViz packages developed by Michael Hahsler.

Features

train association rules
- interactively adjust confidence and support parameters
- sort rules
- sample just top rules to prevent crashes
- post process rules by subsetting LHS or RHS to just variables/items of interest
- suite of interest measures
visualize association rules
- grouped plot, matrix plot, graph, scatterplot, parallel coordinates, item frequency
export association rules to CSV

How to get

Option 1: Copy the code below from the arules_app.R gist

Option2: Source gist directly.

library('devtools')
library('shiny')
library('arules')
library('arulesViz')
source_gist(id='706a28f832a33e90283b')

Option 3: Download the Rsenal package (my personal R package with a hodgepodge of data science tools) and use the arulesApp function:

library('devtools')
install_github('brooksandrew/Rsenal')
library('Rsenal')
?Rsenal::arulesApp

How to use

arulesApp is intended to be called from the R console for interactive and exploratory use. It calls shinyApp which spins up a Shiny app without the overhead of having to worry about placing server.R and ui.R. Calling a Shiny app with a function also has the benefit of smooth passing of parameters and data objects as arguments. More on shinyApp here.

arulesApp is currently highly exploratory (and highly unoptimized). Therefore it works best for quickly iterating on rule training and visualization with low-medium sized datasets. Check out Michael Hahsler’s arulesViz paper for a thorough description of how to interpret the visualizations. There is a particularly useful table on page 24 which compares and summarizes the visualization techniques.

Simply call arulesApp from the console with a data.frame or transaction set for which rules will be mined from:

library('arules') contains Adult and AdultUCI datasets

data('Adult') # transaction set
arulesApp(Adult, vars=40)

data('AdultUCI') # data.frame
arulesApp(AdultUCI)

Here are the arguments:

dataset data.frame, this is the dataset that association rules will be mined from. Each row is treated as a transaction. Seems to work OK when a the S4 transactions class from arules is used, however this is not thoroughly tested.
bin logical, TRUE will automatically discretize/bin numerical data into categorical features that can be used for association analysis.
vars integer, how many variables to include in initial rule mining
supp numeric, the support parameter for initializing visualization. Useful when it is known that a high support is needed to not crash computationally.
conf numeric, the confidence parameter for initializing visualization. Similarly useful when it is known that a high confidence is needed to not crash computationally.

Screenshots

Association rules list view

Scatterplot

Graph

Grouped Plot

Parallel Coordinates

Matrix

Item frequency

Code

Interactive association rules exploration app was originally published by andrew brooks at andrew brooks on November 30, 2015.

Association rule analysis beyond transaction data

2015-11-11T00:00:00+00:00

What is association analysis?
How it works
How to measure rule strength
How to use association analysis with tabular data
When to use association rules with tabular data?
What makes a rule interesting?
Validating rules

Note : I originally prepared this post as an article for Predictive Analytics Times, geared towards a broad audience with a business/practitioner leaning. I ended up writing this article on knitr instead. In a subsequent post, I will discuss the shiny app I created to interactively explore and visualize association rules.

Most analysts think of association rules when dealing with transactional or market basket data. However, association analysis can also provide value with traditional tabular data, especially when little is known about the structure of the data and its domain.

What is association analysis?

Association analysis is an unsupervised learning data mining technique, which at heart is very simple and transparent. The goal is to discover previously unknown patterns (easy) that are also interesting (more challenging) from data. Results are easily interpreted by non-technical audiences and can be expressed in plain English. For example, “When X occurs, Y also occurs 85% of the time.” Many data mining techniques require upfront work in the form of parameter specification and statistical assumption satisfaction before yielding meaningful results. Association analysis is by contrast rather “hands off” upfront. Generating rules is a straightforward procedure that requires little to no knowledge of the underlying data. However, post-processing large rule sets often requires a “hands-on” approach to identify the interesting and valid rules.

Popularized in the early 1990s with this paper from Agrawal et al, association rule analysis has since declined in popularity. I’m still trying to figure out how correlated this decline is with it’s relative usefulness. My conclusion so far is that in most cases it isn’t strong enough to provide much value alone, but can provide value for knowledge discovery in highly exploratory settings. Particularly knowledge discovery of data quirks and structure; knowledge discovery of novel patterns takes more thought.

How it works

The inputs are transactions. In the canonical use-case, these transactions are market basket transactions from a grocery store. For example, 3 transactions could be: {Bread, Milk}, {Bread, Diapers, Beer, Eggs}, {Milk, Diapers, Beer, Cola}. The outputs are association rules that explain the relationship (co-occurrence) between the items. A classic rule example is {Beer} ==> {Diapers} which states that transactions including {Beer} tend to include {Diapers}. A thorough technical description following this example can be found in Kumar’s Introduction to Data Mining textbook.

How to measure rule strength

Association analysis often produces many association rules, more than can be inspected by humans one-by-one. Support and confidence are the most common measures used to initially restrict the number of association rules to just the highest quality. Support measures the relative frequency of the items in a transaction relative to all other transactions. Confidence measures the relative strength of the rule: how often the right hand side (RHS) item occurs in transactions containing the left-hand-side (LHS) item set.

How to use association analysis with tabular data

These same principles can be applied to traditional tabular data (spreadsheets or tables from a relational database). Each row is treated as a transaction. All transactions have the same number of items (one per column). Items are represented by the variable name and the value of that variable for the given row. For example, one row from a five column dataset of students could be represented as: {finalMathGrade=B, eyeColor=brown, state=Minnesota, playSports=YES, inBand=NO}. Traditional association rules run on categorical data. If a table contains continuous data of interest, it can be discretized into bins for analysis.

When to use association rules with tabular data?

I find association analysis to be most handy as a first-step approach when presented with a new dataset. One can think of lists of association rules as lists of facts (with regards to past training data). In market basket analysis problems, these facts can be useful and drive business decisions by themselves. With tabular data, I find association rules more useful as a means to develop a deeper understanding of the data and inform more sophisticated analyses.

1. Learn data structure

This is especially important if you don’t know how your data was generated. Many datasets are products of highly human processes that can appear unintuitive to an unfamiliar analyst. Rules with a perfect confidence of 1 will explain ground truths in your data that are usually indicative of business rules, which would be difficult to uncover without documentation or a domain expert. For example when one column V1 equals ‘A’ or ‘B’, column V5 always equals ‘C’. These rules of confidence 1 often include items where variables are equal to “missing.” Since association rules handles only categorical variables, imputation or exclusion of missing values to accommodate numerical computation is not necessary. Missing values are commonly coded as just another possible category rather than discarded. Rules that include “missing” can always be pruned ex post if they are uninteresting.

2. Learn domain

Association rules are often (and rightly) criticized for generating gobs of trivial and redundant rules. However, rules that are trivial to a seasoned domain expert might provide new and useful information to a data scientist jumping into a new domain (where I often find myself). While often not the final answer, association rules can be used to start the conversation and knowledge sharing process between analysts, domain experts and stakeholders. I often find myself in chicken-egg scenarios where domain experts ask us (the consulting data scientists), the data scientists, what we need to know. I often know so little about the domain and data context, I don’t even know how to start asking intelligent questions. Association rules can be that starting point.

3. Hypothesis generation

This is a trickier endeavor. Discovering obvious patterns using association analysis as a newcomer to a dataset is useful, but relatively easy. Discovering the hidden insights that more directly lead to business decisions is difficult. On a recent project I was asked to discover “unknown unknowns” with little domain context and no specific problem to solve. We employed association rules with heavy amounts of rule post-processing using both human and statistical measures (discussed below) to home in on interesting rules which generated hypotheses to investigate further. Although most of the rules exposed database quality issues rather than “big picture” issues, it was a productive first step.

What makes a rule interesting?

Rules can be objectively interesting (to a computer) and/or subjectively interesting (to a human). Objectively interesting rules are quantified statistically from data by interest measures such as Lift (Interest), Conviction and Leverage. Subjectively interesting rules provide insight that is unexpected and/or actionable to a person. The end goal is obviously to develop rules that are interesting (subjectively) to the user or analyst. Human (especially expert human) time is more expensive than computer time, so objective interest measures are used to first sort and prune massive rule sets. The process of identifying interesting rules is usually iterative, requiring input from domain experts and post-processing adjustments from analysts using quantitative interest measures at each iteration. Visualization helps analysts understand the structure of association rules when there are too many to inspect manually. Michael Hahsler introduces a handy toolkit for visualization in the R arulesViz package, which builds on the arules package for mining rules.

Validating rules

It’s tempting to over-interpret association rules which present seemingly unbiased patterns very matter-of-factly. I experienced this on a recent project. We were searching for previously unknown relationships between software assets by monitoring time intervals where the same software assets consistently experienced issues in production. Initial feedback was stronger than expected, so we increased the weight of association rule relationships in our network model. However, once we began validating our results with software SMEs more thoroughly, we realized many of our rules were spurious. Association rules suffer from the Vast Search Effect: the tendency to discover spurious patterns when considering many possibilities. Researchers have proposed techniques to mitigate the issue, such as Swap Randomization and post-processing pruning. However, adoption in popular statistical software has been slow to catch on. For this reason, association analysis can be a dangerous approach to produce “final answers.” Instead, consider association analysis as a tool for exploratory data analysis to get familiar with a dataset’s structure, its domain and generate interesting questions to pursue further with more sophisticated analysis.

Association rule analysis beyond transaction data was originally published by andrew brooks at andrew brooks on November 11, 2015.

Advanced tips and tricks with data.table

2015-08-31T00:00:00+00:00

1. DATA STRUCTURES & ASSIGNMENT
2. BY
3. FUNCTIONS
- Passing data.table column names as function arguments
  - Method 2: quotes and get
- Beware of scoping within data.table
  - data.frame way
  - data.table way
4. PRINTING
- Print data.table with []
- Hide output from := with knitr

Tips and tricks learned along the way

This is mostly a running list of data.table tricks that took me a while to figure out either by digging into the official documentation, adapting StackOverflow posts, or more often than not, experimenting for hours. I’d like to persist these discoveries somewhere with more memory than my head (hello internet) so I can reuse them after my mental memory forgets them. A less organized and concise addition to DataCamp’s sweet cheat sheet for the basics.

Most, if not all of these techniques were developed for real data science projects and provided some value to my data engineering. I’ve generalized everything to the mtcars dataset which might not make this value immediately clear in this slightly contrived context. This list is not intended to be comprehensive as DataCamp’s data.table cheatsheet is. OK, enough disclaimers!

Some more advanced functionality from data.table creator Matt Dowle here.

1. DATA STRUCTURES & ASSIGNMENT

Columns of lists

summary table (long and narrow)

This could be useful, but is easily achievable using traditional methods.

dt <- data.table(mtcars)[, .(cyl, gear)]
dt[,unique(gear), by=cyl]

##    cyl V1
## 1:   6  4
## 2:   6  3
## 3:   6  5
## 4:   4  4
## 5:   4  3
## 6:   4  5
## 7:   8  3
## 8:   8  5

summary table (short and narrow)

Add all categories of gear for each cyl to original data.table as a list.

This is more nifty. It’s so simple, I find myself using this trick to quickly explore data ad hoc at the command line. Can also be useful for more serious data engineering.

dt <- data.table(mtcars)[,.(gear, cyl)]
dt[,gearsL:=list(list(unique(gear))), by=cyl] # original, ugly
dt[,gearsL:=.(list(unique(gear))), by=cyl] # improved, pretty
head(dt)

##    gear cyl gearsL
## 1:    4   6  4,3,5
## 2:    4   6  4,3,5
## 3:    4   4  4,3,5
## 4:    3   6  4,3,5
## 5:    3   8    3,5
## 6:    3   6  4,3,5

Update 10/29/2015: Per these comments on StackOverlow referencing my post, t[,gearsL:=list(list(unique(gear))), by=cyl] can be more elegantly written as t[,gearsL:=.(list(unique(gear))), by=cyl]. Thanks for pointing out my unnecessarily verbose and unusual syntax! I think I wrote the first thing that worked when I posted this, not realizing the normal .( syntax was equivalent to the outer list.

Accessing elements from a column of lists

Extract second element of each list in gearL1 and create row gearL1. This isn’t that groundbreaking, but explores how to access elements of columns which are constructed of lists of lists. lapply is your friend.

dt[,gearL1:=lapply(gearsL, function(x) x[2])]
dt[,gearS1:=sapply(gearsL, function(x) x[2])] 

head(dt)

##    gear cyl gearsL gearL1 gearS1
## 1:    4   6  4,3,5      3      3
## 2:    4   6  4,3,5      3      3
## 3:    4   4  4,3,5      3      3
## 4:    3   6  4,3,5      3      3
## 5:    3   8    3,5      5      5
## 6:    3   6  4,3,5      3      3

str(head(dt[,gearL1]))

## List of 6
##  $ : num 3
##  $ : num 3
##  $ : num 3
##  $ : num 3
##  $ : num 5
##  $ : num 3

str(head(dt[,gearS1]))

##  num [1:6] 3 3 3 3 5 3

Update 9/24/2015: Per Matt Dowle’s comments, a slightly more syntactically succinct way of doing this:

dt[,gearL1:=lapply(gearsL, `[`, 2)]
dt[,gearS1:=sapply(gearsL, `[`, 2)]

Calculate all the gears for all cars of each cyl (excluding the current current row). This can be useful for comparing observations to the mean of groups, where the group mean is not biased by the observation of interest.

dt[,other_gear:=mapply(function(x, y) setdiff(x, y), x=gearsL, y=gear)]
head(dt)

##    gear cyl gearsL gearL1 gearS1 other_gear
## 1:    4   6  4,3,5      3      3        3,5
## 2:    4   6  4,3,5      3      3        3,5
## 3:    4   4  4,3,5      3      3        3,5
## 4:    3   6  4,3,5      3      3        4,5
## 5:    3   8    3,5      5      5          5
## 6:    3   6  4,3,5      3      3        4,5

Update 9/24/2015: Per Matt Dowle’s comments, this achieves the same as above.

dt[,other_gear:=mapply(setdiff, gearsL, gear)]

Suppressing intermediate output with {}

This is actually a base R trick that I didn’t discover until working with data.table. See ?`{` for some documentation and examples. I’ve only used it within the J slot of data.table, it might be more generalizable. I find it pretty useful for generating columns on the fly when I need to perform some multi-step vectorized operation. It can clean up code by allowing you to reference the same temporary variable by a concise name rather than rewriting the code to re-compute it.

dt <- data.table(mtcars)

Defaults to just returning the last object defined in the braces unnamed.

dt[,{tmp1=mean(mpg); tmp2=mean(abs(mpg-tmp1)); tmp3=round(tmp2, 2)}, by=cyl]

##    cyl   V1
## 1:   6 1.19
## 2:   4 3.83
## 3:   8 1.79

We can be more explicit by passing a named list of what we want to keep.

dt[,{tmp1=mean(mpg); tmp2=mean(abs(mpg-tmp1)); tmp3=round(tmp2, 2); list(tmp2=tmp2, tmp3=tmp3)}, by=cyl]

##    cyl     tmp2 tmp3
## 1:   6 1.191837 1.19
## 2:   4 3.833058 3.83
## 3:   8 1.785714 1.79

Can also write it like this without semicolons.

dt[,{tmp1=mean(mpg)
     tmp2=mean(abs(mpg-tmp1))
     tmp3=round(tmp2, 2)
     list(tmp2=tmp2, tmp3=tmp3)},
   by=cyl]

##    cyl     tmp2 tmp3
## 1:   6 1.191837 1.19
## 2:   4 3.833058 3.83
## 3:   8 1.785714 1.79

This is trickier with := assignments… I don’t think := is intended to work when wrapped in {. Assigning multiple columns with := at once does not allow you to use the first columns you create to use building the ones after it, as we did with = inside the { above. Chaining and then dropping unwanted variables is a messy workaround… still exploring this one.

dt <- data.table(mtcars)[,.(cyl, mpg)]

dt[,tmp1:=mean(mpg), by=cyl][,tmp2:=mean(abs(mpg-tmp1)), by=cyl][,tmp1:=NULL]
head(dt)

##    cyl  mpg     tmp2
## 1:   6 21.0 1.191837
## 2:   6 21.0 1.191837
## 3:   4 22.8 3.833058
## 4:   6 21.4 1.191837
## 5:   8 18.7 1.785714
## 6:   6 18.1 1.191837

Fast looping with `set`

I still haven’t worked much with the loop + set framework. I’ve been able to achieve pretty much everything with := which is more flexible and powerful. However, if you must loop, set is orders of magnitude faster than native R assignments within loops. Here’s a snippet from data.table news a while back:

New function set(DT,i,j,value) allows fast assignment to elements
of DT. Similar to := but avoids the overhead of [.data.table, so is
much faster inside a loop. Less flexible than :=, but as flexible
as matrix sub-assignment. Similar in spirit to setnames(), setcolorder(),
setkey() and setattr(); i.e., assigns by reference with no copy at all.

M = matrix(1,nrow=100000,ncol=100)
DF = as.data.frame(M)
DT = as.data.table(M)
system.time(for (i in 1:1000) DF[i,1L] <- i)   # 591.000s
system.time(for (i in 1:1000) DT[i,V1:=i])     #   1.158s
system.time(for (i in 1:1000) M[i,1L] <- i)    #   0.016s
system.time(for (i in 1:1000) set(DT,i,1L,i))  #   0.027s

data.table creators do favor set for some things, like this task which can also be done w/ lapply and .SD. I was actually directed to this solution after I posed this question on StackOverflow. I was also pleased to learn that the functionality I was looking for – applying a function to a subset of columns with .SDcols while preserving the untouched columns – was added as a feature request.

dt <- data.table(mtcars)[,1:5, with=F]
for (j in c(1L,2L,4L)) set(dt, j=j, value=-dt[[j]]) # integers using 'L' passed for efficiency
for (j in c(3L,5L)) set(dt, j=j, value=paste0(dt[[j]],'!!'))
head(dt)

##      mpg cyl  disp   hp   drat
## 1: -21.0  -6 160!! -110  3.9!!
## 2: -21.0  -6 160!! -110  3.9!!
## 3: -22.8  -4 108!!  -93 3.85!!
## 4: -21.4  -6 258!! -110 3.08!!
## 5: -18.7  -8 360!! -175 3.15!!
## 6: -18.1  -6 225!! -105 2.76!!

Using `shift` for to lead/lag vectors and lists

Note this feature is only available in version 1.9.5 (currently on Github, not CRAN) Base R surprisingly does not have great tools for dealing with leads/lags of vectors that most social science statistical software (Stata, SAS, even FAME which I used in my formative data years) come equipped with out of the box.

dt <- data.table(mtcars)[,.(mpg, cyl)]
dt[,mpg_lag1:=shift(mpg, 1)]
dt[,mpg_forward1:=shift(mpg, 1, type='lead')]
head(dt)

##     mpg cyl mpg_lag1 mpg_forward1
## 1: 21.0   6       NA         21.0
## 2: 21.0   6     21.0         22.8
## 3: 22.8   4     21.0         21.4
## 4: 21.4   6     22.8         18.7
## 5: 18.7   8     21.4         18.1
## 6: 18.1   6     18.7         14.3

`shift` with `by`

# creating some data
n <- 30
dt <- data.table(
  date=rep(seq(as.Date('2010-01-01'), as.Date('2015-01-01'), by='year'), n/6), 
  ind=rpois(n, 5),
  entity=sort(rep(letters[1:5], n/5))
  )

setkey(dt, entity, date) # important for ordering
dt[,indpct_fast:=(ind/shift(ind, 1))-1, by=entity]

lagpad <- function(x, k) c(rep(NA, k), x)[1:length(x)] 
dt[,indpct_slow:=(ind/lagpad(ind, 1))-1, by=entity]

head(dt, 10)

##  1: 2010-01-01   3      a          NA          NA
##  2: 2011-01-01   2      a  -0.3333333  -0.3333333
##  3: 2012-01-01   5      a   1.5000000   1.5000000
##  4: 2013-01-01   4      a  -0.2000000  -0.2000000
##  5: 2014-01-01   1      a  -0.7500000  -0.7500000
##  6: 2015-01-01   5      a   4.0000000   4.0000000
##  7: 2010-01-01   2      b          NA          NA
##  8: 2011-01-01   6      b   2.0000000   2.0000000
##  9: 2012-01-01   8      b   0.3333333   0.3333333
## 10: 2013-01-01   9      b   0.1250000   0.1250000

Create multiple columns with `:=` in one statement

This is useful, but note that that the columns operated on must be atomic vectors or lists. That is they must exist before running computation.
Building columns referencing other columns in this set need to be done individually or chained.

dt <- data.table(mtcars)[,.(mpg, cyl)]
dt[,`:=`(avg=mean(mpg), med=median(mpg), min=min(mpg)), by=cyl]
head(dt)

##     mpg cyl      avg  med  min
## 1: 21.0   6 19.74286 19.7 17.8
## 2: 21.0   6 19.74286 19.7 17.8
## 3: 22.8   4 26.66364 26.0 21.4
## 4: 21.4   6 19.74286 19.7 17.8
## 5: 18.7   8 15.10000 15.2 10.4
## 6: 18.1   6 19.74286 19.7 17.8

Assign a column with `:=` named with a character object

This is the advised way to assign a new column whose name you already have determined and saved as a character. Simply surround the character object in parentheses.

dt <- data.table(mtcars)[, .(cyl, mpg)]

thing2 <- 'mpgx2'
dt[,(thing2):=mpg*2]

head(dt)

##    cyl  mpg mpgx2
## 1:   6 21.0  42.0
## 2:   6 21.0  42.0
## 3:   4 22.8  45.6
## 4:   6 21.4  42.8
## 5:   8 18.7  37.4
## 6:   6 18.1  36.2

This is old (now deprecated) way which still works for now. Not advised.

thing3 <- 'mpgx3'
dt[,thing3:=mpg*3, with=F]

head(dt)

##    cyl  mpg mpgx2 mpgx3
## 1:   6 21.0  42.0  63.0
## 2:   6 21.0  42.0  63.0
## 3:   4 22.8  45.6  68.4
## 4:   6 21.4  42.8  64.2
## 5:   8 18.7  37.4  56.1
## 6:   6 18.1  36.2  54.3

2. `BY`

Calculate a function over a group (using `by`) excluding each entity in a second category.

This title probably doesn’t immediately make much sense. Let me explain what I’m going to calculate and why with an example. We want to compare the mpg of each car to the average mpg of cars in the same class (the same # of cylinders). However, we don’t want to bias the group mean by including the car we want to compare to the average in that average.

This assumption doesn’t appear useful in this example, but assume that gear+cyl uniquely identify the cars. In the real project where I faced this problem, I was calculating an indicator related to an appraiser relative to the average of all other appraisers in their zip3. (cyl was really zipcode and gear was the appraiser’s ID).

METHOD 1: in-line

0.a Biased mean: simple mean by `cyl`

However we want to know for each row, what is the mean among all the other cars with the same # of cyls, excluding that car.

dt <- data.table(mtcars)[,.(cyl, gear, mpg)]
dt[, mpg_biased_mean:=mean(mpg), by=cyl] 
head(dt)

##    cyl gear  mpg mpg_biased_mean
## 1:   6    4 21.0        19.74286
## 2:   6    4 21.0        19.74286
## 3:   4    4 22.8        26.66364
## 4:   6    3 21.4        19.74286
## 5:   8    3 18.7        15.10000
## 6:   6    3 18.1        19.74286

1.a `.GRP` without setting key

dt[, dt[!gear %in% unique(dt$gear)[.GRP], mean(mpg), by=cyl], by=gear] #unbiased mean

##    gear cyl       V1
## 1:    4   6 19.73333
## 2:    4   8 15.10000
## 3:    4   4 25.96667
## 4:    3   6 19.74000
## 5:    3   4 27.18000
## 6:    3   8 15.40000
## 7:    5   6 19.75000
## 8:    5   4 26.32222
## 9:    5   8 15.05000

# check
dt[gear!=4 & cyl==6, mean(mpg)]

## [1] 19.73333

Update 9/24/2015: Per Matt Dowle’s comments, this also works with slightly less code. For my simple example, there was also a marginal speed gain. Time savings relative to the .GRP method will likely increase with the complexity of the problem.

dt[, dt[!gear %in% .BY[[1]], mean(mpg), by=cyl], by=gear] #unbiased mean

##    gear cyl       V1
## 1:    4   6 19.73333
## 2:    4   8 15.10000
## 3:    4   4 25.96667
## 4:    3   6 19.74000
## 5:    3   4 27.18000
## 6:    3   8 15.40000
## 7:    5   6 19.75000
## 8:    5   4 26.32222
## 9:    5   8 15.05000

1.b Same as 1.a, but a little faster

uid <- unique(dt$gear)
dt[, dt[!gear %in% (uid[.GRP]), mean(mpg), by=cyl] , by=gear][order(cyl, gear)] #unbiased mean

##    gear cyl       V1
## 1:    3   4 27.18000
## 2:    4   4 25.96667
## 3:    5   4 26.32222
## 4:    3   6 19.74000
## 5:    4   6 19.73333
## 6:    5   6 19.75000
## 7:    3   8 15.40000
## 8:    4   8 15.10000
## 9:    5   8 15.05000

Why does this work?

# 1.a pulling it apart with .GRP
dt[, .GRP, by=cyl]

##    cyl GRP
## 1:   6   1
## 2:   4   2
## 3:   8   3

dt[, .(.GRP, unique(dt$gear)[.GRP]), by=cyl]

##    cyl GRP V2
## 1:   6   1  4
## 2:   4   2  3
## 3:   8   3  5

dt[,dt[, .(.GRP, unique(dt$gear)[.GRP]), by=cyl], by=gear]

##    gear cyl GRP V2
## 1:    4   6   1  4
## 2:    4   4   2  3
## 3:    4   8   3  5
## 4:    3   6   1  4
## 5:    3   4   2  3
## 6:    3   8   3  5
## 7:    5   6   1  4
## 8:    5   4   2  3
## 9:    5   8   3  5

1.b Setting key

setkey(dt, gear)
uid <- unique(dt$gear)
dt[, dt[!.(uid[.GRP]), mean(mpg), by=cyl] , by=gear] #unbiased mean

##    gear cyl       V1
## 1:    3   6 19.74000
## 2:    3   4 27.18000
## 3:    3   8 15.40000
## 4:    4   6 19.73333
## 5:    4   8 15.10000
## 6:    4   4 25.96667
## 7:    5   6 19.75000
## 8:    5   8 15.05000
## 9:    5   4 26.32222

mean(dt[cyl==4 & gear!=3,mpg]) # testing

## [1] 27.18

mean(dt[cyl==6 & gear!=3,mpg]) # testing

## [1] 19.74

METHOD 2: using `{}` and `.SD`

{} is used for to suppress intermediate operations.

Building up

No surprises here.

dt[,  .SD[, mean(mpg)], by=gear] # same as `dt[, mean(mpg), by=gear]`

##    gear       V1
## 1:    3 16.10667
## 2:    4 24.53333
## 3:    5 21.38000

dt[,  .SD[, mean(mpg), by=cyl], by=gear] # same as `dt[, mean(mpg), by=.(cyl, by=gear)]`

##    gear cyl     V1
## 1:    3   6 19.750
## 2:    3   8 15.050
## 3:    3   4 21.500
## 4:    4   6 19.750
## 5:    4   4 26.925
## 6:    5   4 28.200
## 7:    5   8 15.400
## 8:    5   6 19.700

Nested data.tables and `by` statements

This chunk shows what happens with two by statements nested within two different data.tables. Explanatory purposes only - not necessary for our task. n counts the # of cars in that cyl. N counts the number of cars by cyl and gear.

dt[,{
  vbar = sum(mpg)
  n = .N
  .SD[,.(n, .N, sum_in_gear_cyl=sum(mpg), sum_in_cyl=vbar), by=gear]
} , by=cyl]

##    cyl gear  n  N sum_in_gear_cyl sum_in_cyl
## 1:   6    3  7  2            39.5      138.2
## 2:   6    4  7  4            79.0      138.2
## 3:   6    5  7  1            19.7      138.2
## 4:   8    3 14 12           180.6      211.4
## 5:   8    5 14  2            30.8      211.4
## 6:   4    3 11  1            21.5      293.3
## 7:   4    4 11  8           215.4      293.3
## 8:   4    5 11  2            56.4      293.3

dt[,sum(mpg), by=cyl] # test

##    cyl    V1
## 1:   6 138.2
## 2:   8 211.4
## 3:   4 293.3

Calculating “unbiased mean”

This is in a summary table. This would need to be merged back onto dt if that is desired.

dt[,{
  vbar = mean(mpg)
  n = .N
  .SD[,(n*vbar-sum(mpg))/(n-.N),by=gear]
} , by=cyl]

##    cyl gear       V1
## 1:   6    3 19.74000
## 2:   6    4 19.73333
## 3:   6    5 19.75000
## 4:   8    3 15.40000
## 5:   8    5 15.05000
## 6:   4    3 27.18000
## 7:   4    4 25.96667
## 8:   4    5 26.32222

METHOD 3: Super Fast Mean calculation

Non-function direct way

Using a vectorized approach to calculate the unbiased mean for each combination of gear and cyl. Mechanically, it calculates the “biased average” for all cars by cyl. Then subtract off the share of cars with the combination of gear and cyl that we want to exclude from the average and add that share. Then extrapolate out this pared down mean.

dt <- data.table(mtcars)[,.(mpg,cyl,gear)]
dt[,`:=`(avg_mpg_cyl=mean(mpg), Ncyl=.N), by=cyl]
dt[,`:=`(Ncylgear=.N, avg_mpg_cyl_gear=mean(mpg)), by=.(cyl, gear)]
dt[,unbmean:=(avg_mpg_cyl*Ncyl-(Ncylgear*avg_mpg_cyl_gear))/(Ncyl-Ncylgear)]
setkey(dt, cyl, gear)  
head(dt)

##     mpg cyl gear avg_mpg_cyl Ncyl Ncylgear avg_mpg_cyl_gear  unbmean
## 1: 21.5   4    3    26.66364   11        1           21.500 27.18000
## 2: 22.8   4    4    26.66364   11        8           26.925 25.96667
## 3: 24.4   4    4    26.66364   11        8           26.925 25.96667
## 4: 22.8   4    4    26.66364   11        8           26.925 25.96667
## 5: 32.4   4    4    26.66364   11        8           26.925 25.96667
## 6: 30.4   4    4    26.66364   11        8           26.925 25.96667

Wrapping up code below into a function

leaveOneOutMean <- function(dt, ind, bybig, bysmall) {
  dtmp <- copy(dt) # copy so as not to alter original dt object w intermediate assignments
  dtmp <- dtmp[is.na(get(ind))==F,]
  dtmp[,`:=`(avg_ind_big=mean(get(ind)), Nbig=.N), by=.(get(bybig))]
  dtmp[,`:=`(Nbigsmall=.N, avg_ind_big_small=mean(get(ind))), by=.(get(bybig), get(bysmall))]
  dtmp[,unbmean:=(avg_ind_big*Nbig-(Nbigsmall*avg_ind_big_small))/(Nbig-Nbigsmall)]
  return(dtmp[,unbmean])
}

dt <- data.table(mtcars)[,.(mpg,cyl,gear)]
dt[,unbiased_mean:=leaveOneOutMean(.SD, ind='mpg', bybig='cyl', bysmall='gear')]
dt[,biased_mean:=mean(mpg), by=cyl]
head(dt)

##     mpg cyl gear unbiased_mean biased_mean
## 1: 21.0   6    4      19.73333    19.74286
## 2: 21.0   6    4      19.73333    19.74286
## 3: 22.8   4    4      25.96667    26.66364
## 4: 21.4   6    3      19.74000    19.74286
## 5: 18.7   8    3      15.40000    15.10000
## 6: 18.1   6    3      19.74000    19.74286

Speed check

Method 3 is roughly 100x faster than the other two. Great for this narrow task with the vectorization built in, but less generalizable; The other two methods allow any function to be passed.

dt <- data.table(mtcars)
dt <- dt[sample(1:.N, 100000, replace=T), ] # increase # of rows in mtcars
dt$gear <- sample(1:300, nrow(dt), replace=T) # adding in more cateogries

Method 3:

system.time(dt[,unbiased_mean_vectorized:=leaveOneOutMean(.SD, ind='mpg', bybig='cyl', bysmall='gear')])

##    user  system elapsed 
##   0.033   0.003   0.035

Method 2:

system.time(tmp <- dt[,dt[!gear %in% unique(dt$gear)[.GRP], mean(mpg), by=cyl], by=gear] )

##    user  system elapsed 
##   3.709   0.359   4.069

Method 1:

uid <- unique(dt$gear)
system.time(dt[, dt[!gear %in% (uid[.GRP]), mean(mpg), by=cyl] , by=gear][order(cyl, gear)])

##    user  system elapsed 
##   3.345   0.331   3.677

`keyby` to key resulting aggregate table

Without `keyby`

Categories are not sorted

## devtools::install_github('brooksandrew/Rsenal')
library('Rsenal') # grabbing depthbin function
tmp <- dt[, .(N=.N, sum=sum(vs), mean=mean(vs)/.N), by=depthbin(mpg, 5, labelOrder=T)]
tmp

##           depthbin     N   sum         mean
## 1: (15.2,17.8] 2/5 15372  3131 1.325020e-05
## 2:   (17.8,21] 3/5 21839  6204 1.300787e-05
## 3: [10.4,15.2] 1/5 25255     0 0.000000e+00
## 4:   (21,24.4] 4/5 18817 18817 5.314343e-05
## 5: (24.4,33.9] 5/5 18717 15581 4.447571e-05

tmp[,barplot(mean, names=depthbin, las=2)]

##      [,1]
## [1,]  0.7
## [2,]  1.9
## [3,]  3.1
## [4,]  4.3
## [5,]  5.5

With `keyby`

## devtools::install_github('brooksandrew/Rsenal')
library('Rsenal')
tmp <- dt[, .(N=.N, sum=sum(vs), mean=mean(vs)/.N), keyby=depthbin(mpg, 5, labelOrder=T)]
tmp

##           depthbin     N   sum         mean
## 1: [10.4,15.2] 1/5 25255     0 0.000000e+00
## 2: (15.2,17.8] 2/5 15372  3131 1.325020e-05
## 3:   (17.8,21] 3/5 21839  6204 1.300787e-05
## 4:   (21,24.4] 4/5 18817 18817 5.314343e-05
## 5: (24.4,33.9] 5/5 18717 15581 4.447571e-05

tmp[,barplot(mean, names=depthbin, las=2)]

##      [,1]
## [1,]  0.7
## [2,]  1.9
## [3,]  3.1
## [4,]  4.3
## [5,]  5.5

Using `[1]`, `[.N]`, `setkey` and `by` for within group subsetting

take highest value of column A when column B is highest by group

Max of qsec for each category of cyl (this is easy)

dt <- data.table(mtcars)[, .(cyl, mpg, qsec)]
dt[, max(qsec), by=cyl]

##    cyl    V1
## 1:   6 20.22
## 2:   4 22.90
## 3:   8 18.00

value of `qsec` when `mpg` is the highest per category of `cyl`

(this is trickier)

setkey(dt, mpg)
dt[,qsec[.N],  by=cyl]

##    cyl    V1
## 1:   8 17.05
## 2:   6 19.44
## 3:   4 19.90

value of `qsec` when `mpg` is the lowest per category of `cyl`

dt[,qsec[1],  by=cyl]

##    cyl    V1
## 1:   8 17.98
## 2:   6 18.90
## 3:   4 18.60

value of `qsec` when `mpg` is the median per category of `cyl`

dt[,qsec[round(.N/2)],  by=cyl]

##    cyl   V1
## 1:   8 18.0
## 2:   6 15.5
## 3:   4 16.7

subset rows within by statement

V1 is the standard deviation of mpg by cyl. V2 is the standard deviation of mpg for just the first half of mpg.

dt <- data.table(mtcars)
setkey(dt,mpg)
dt[, .(sd(mpg), sd(mpg[1:round(.N/2)])), by=cyl]

##    cyl       V1        V2
## 1:   8 2.560048 2.0926174
## 2:   6 1.453567 0.8981462
## 3:   4 4.509828 1.7728508

3. FUNCTIONS

Passing `data.table` column names as function arguments

Method 1: No quotes, and `deparse` + `substitute`

This way seems more data.table-ish because it maintains the practice of not using quotes on variable names in most cases.

dt <- data.table(mtcars)[,.(cyl, mpg)]
myfunc <- function(dt, v) {
  v2=deparse(substitute(v))
  dt[,v2, with=F][[1]] # [[1]] returns a vector instead of a data.table
}

myfunc(dt, mpg)

##  [1] 21.0 21.0 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 17.8 16.4 17.3 15.2
## [15] 10.4 10.4 14.7 32.4 30.4 33.9 21.5 15.5 15.2 13.3 19.2 27.3 26.0 30.4
## [29] 15.8 19.7 15.0 21.4

Method 2: quotes and `get`

However I tend to pass through column names as characters (quoted) and use get each time I reference that column. That can be annoying if you have a long function repeatedly reference column names, but I often need to write such few lines of code with data.table, it hasn’t struck me as terribly unslick, yet.

dt <- data.table(mtcars)
myfunc <- function(dt, v) dt[,get(v)]

myfunc(dt, 'mpg')

##  [1] 21.0 21.0 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 17.8 16.4 17.3 15.2
## [15] 10.4 10.4 14.7 32.4 30.4 33.9 21.5 15.5 15.2 13.3 19.2 27.3 26.0 30.4
## [29] 15.8 19.7 15.0 21.4

Beware of scoping within data.table

`data.frame` way

When you add something to a data.frame within a function that exists in the global environment, it does not affect that object in the global environment unless you return and reassign it as such, or you use the <<- operator.

df <- mtcars[,c('cyl', 'mpg')]
add_column_df <- function(df) {
  df$addcol1<- 'here in func!'
  df$addcol2 <<- 'in glob env!'
  return(df)
}

When we call the function, we see addcol1 in the output. But not addcol2. That’s because it’s been added to the df in the global environment one level up.

head(add_column_df(df))

##                   cyl  mpg       addcol1
## Mazda RX4           6 21.0 here in func!
## Mazda RX4 Wag       6 21.0 here in func!
## Datsun 710          4 22.8 here in func!
## Hornet 4 Drive      6 21.4 here in func!
## Hornet Sportabout   8 18.7 here in func!
## Valiant             6 18.1 here in func!

Here is addcol2, but not addcol.

head(df)

##                   cyl  mpg      addcol2
## Mazda RX4           6 21.0 in glob env!
## Mazda RX4 Wag       6 21.0 in glob env!
## Datsun 710          4 22.8 in glob env!
## Hornet 4 Drive      6 21.4 in glob env!
## Hornet Sportabout   8 18.7 in glob env!
## Valiant             6 18.1 in glob env!

`data.table` way

Unlike data.frame, the := operator adds a column to both the object living in the global environment and used in the function. I think this is because these objects are actually the same object. data.table shaves computation time by not making copies unless explicitly directed to.

dt <- data.table(mtcars)
add_column_dt <- function(dat) {
  dat[,addcol:='sticking_to_dt!'] # hits dt in glob env
  return(dat)
}
head(add_column_dt(dt)) # addcol here

##     mpg cyl disp  hp drat    wt  qsec vs am gear carb          addcol
## 1: 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4 sticking_to_dt!
## 2: 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4 sticking_to_dt!
## 3: 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1 sticking_to_dt!
## 4: 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1 sticking_to_dt!
## 5: 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2 sticking_to_dt!
## 6: 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1 sticking_to_dt!

head(dt) # addcol also here

##     mpg cyl disp  hp drat    wt  qsec vs am gear carb          addcol
## 1: 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4 sticking_to_dt!
## 2: 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4 sticking_to_dt!
## 3: 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1 sticking_to_dt!
## 4: 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1 sticking_to_dt!
## 5: 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2 sticking_to_dt!
## 6: 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1 sticking_to_dt!

So something like this renaming the local version using copy bypasses this behavior, but is likely somewhat less efficient (and elegant). I suspect there’s a cleaner and/or faster way to do this: keep some variables local to the function while persisting and returning other columns.

dt <- data.table(mtcars)
add_column_dt <- function(dat) {
  datloc <- copy(dat)
  datloc[,addcol:='not sticking_to_dt!'] # hits dt in glob env
  return(datloc)
}
head(add_column_dt(dt)) # addcol here

##     mpg cyl disp  hp drat    wt  qsec vs am gear carb              addcol
## 1: 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4 not sticking_to_dt!
## 2: 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4 not sticking_to_dt!
## 3: 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1 not sticking_to_dt!
## 4: 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1 not sticking_to_dt!
## 5: 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2 not sticking_to_dt!
## 6: 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1 not sticking_to_dt!

head(dt) # addcol not here

##     mpg cyl disp  hp drat    wt  qsec vs am gear carb
## 1: 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## 2: 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## 3: 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## 4: 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## 5: 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## 6: 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

4. PRINTING

Print data.table with `[]`

Nothing groundbreaking here, but a small miscellaneous piece of functionality. In data.frame world, wrapping an expression in () prints the output to the console. This also works with data.table, but there is another way. In data.table this is achieved by appending [] to the end of the expression. I find this useful because when I’m exploring at the console, I don’t usually decide to print the output until I’m almost done and I’m already at the end of the expression I’ve written.

# data.frame way of printing after an assignment
df <- head(mtcars) # doesn't print
(df <- head(mtcars)) # does print

##                    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

# data.table way of printing after an assignment
dt <- data.table(head(mtcars)) # doesn't print
dt[,hp2wt:=hp/wt][] # does print

##     mpg cyl disp  hp drat    wt  qsec vs am gear carb    hp2wt
## 1: 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4 41.98473
## 2: 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4 38.26087
## 3: 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1 40.08621
## 4: 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1 34.21462
## 5: 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2 50.87209
## 6: 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1 30.34682

Hide output from `:=` with knitr

It used to be that assignments using the := operator printed the object to console when knitting documents with knitr and rmarkdown. This is actually fixed in data.table v1.9.5. However at the time of my writing, this currently not available on CRAN… only Github. For 1.9.4 users, this StackOverflow post has some hacky solutions. This least impedance approach I found was simply wrapping the expression in invisible. Other solutions alter the way you use data.table which I didn’t like.

dt <- data.table(mtcars)
dt[,mpg2qsec:=mpg/qsec] # will print with knitr
invisible(dt[,mpg2qsec:=mpg/qsec]) # won't print with knitr

Advanced tips and tricks with data.table was originally published by andrew brooks at andrew brooks on August 31, 2015.

Render reports directly from R scripts

2015-03-05T00:00:00+00:00

Workflow

This post is really about workflow. Specifically a data-science workflow, although it should be relevant for others. It will probably resonate most (if at all) with those who have some experience (mostly positive) generating reports from Rmarkdown files with knitr, but might have some gripes. Maybe not gripes, maybe just feelings of uncertainty over whether it makes sense to contain your hard work in an Rmarkdown file or an R script, or both.

Generate reports with Rmarkdown (Rmd) files

With Rmarkdown, you can generate these stylish reports with code like this.

Generate reports directly from R scripts

One can also cut out the middle-man (Rmd) and generate the exact same HTML, PDF and Word reports using native R scripts. This was news to me until this week. It’s a subtle difference, but one that I’ve found nimble and powerful in all the right places. Check this out for a quick intro.

How it works: Code as normal. Tweak the comments in your code to render the document text, headers, format, style, etc. of your report however you like. You can compile any old R script, regardless of it’s structure, but there are a lot of options at your disposal for formatting and prettifying, if that’s your thing. Then it’s a one liner to compile into a report:

library('rmarkdown')
rmarkdown::render('/Users/you/Documents/yourscript.R')

Rmarkdown vs R

Rmd != R: You can’t source an Rmarkdown file like you would an R script. I have no doubt there are tools that exist (or can be easily developed) to strip the code chunks from an Rmarkdown file, but this seems cumbersome.
Competing incentives: presentation vs. workflow: When you’ve got tons of code chunks with just a few lines each, it can be annoying to test your code without knitting (compiling) your entire document. I often purposely keep chunks big to facilitate running blocks of selected code interactively. This makes for smooth coding, but slightly more obtuse documents. One strategy I’ve tried is to “Rmarkdownify” my code only after I’ve thoroughly developed and tested it… but then when it comes time to re-examine, change or pipe code someplace else, you’ve got this Rmarkdown document to overhaul. And in my work (many more parts analysis than development), I’m rarely ever done or know when I’m done.
No need to duplicate Rmd and R scripts: Say you’re writing some data wrangling code that pulls from a handful of data sources, merges them all together, aggregates, scales and transforms them into an analytics ready dataset. You want to document this process… but you also want to be able to pipe this piece of ETL code elsewhere. I’ve been tempted in the past to maintain both a bare-bones R script and a verbose flowery Rmd file describing the process. This keeps both the developers (on your team or within yourself) happy and the consumers of your analysis happy… but it will probably drive you crazy maintaining two versions of more-or-less the same thing. With an R script formatted with markdown-style comments, you might be able to get the two birds with one stone.
Run-time: This isn’t very well addressed by either method, but I certainly find it easier to work with bigger data anything computationally intensive using native R scripts. When I knit a big Rmarkdown script, I often cross my fingers and hope it doesn’t bug 95% through and I have to start over. By default, knitting .Rmd files does not persist objects to the Global Environment, although I’d be surprised if there wasn’t a way to change this.
All pros, no cons: If you’re working on a team that doesn’t want to use knitr and Rmarkdown, no matter. Your team members might gaze at seemingly strange comments in your R scripts, but they can run, read, edit and pipe your code as if it was their own. You can even compile their code into reports. This will essentially just separate code from output and plots printed to the console. It might not be the prettiest, but it sure beats saving off graphics and results and copying and pasting into slides somewhere. And I find it’s easier to find your chart, finding, or what-have-you in a compiled document than within a script where you have to run code, dependencies and likely muddle up the current environment in which you’re working.

Rendered report in the flesh

All the features I’m used to using with Rmarkdown documents worked when embedded in native R scripts.

The script below (also here) generates this html document (below).

HTML report generated by R script below

R script that generates the html report above

This is perhaps not a great example of how a typical R script would look. A typical R script/document would probably have significantly more code and less comments. However, I know how code appears in a report – my purpose is really to test the markdown functionality.

#' ---
#' title: Sample HTML report generated from R script
#' author: Andrew Brooks
#' date: March 4, 2015
#' output:
#'    html_document:
#'      toc: true
#'      highlight: zenburn
#' ---

#' ## Generate document body from comments
#' All the features from markdown and markdown supported within .Rmd documents, I was able to
#' get from within R scripts.  Here are some that I tested and use most frequently:  
#' 
#' * Smart comment fomatting in your R script generate the body and headers of the document
#'     * Simply tweak your comments to begin with `#'` instead of just `#`  
#' * Create markdown headers as normal: `#' #` for h1, `#' ##` for h2, etc.
#' * Add two spaces to the end of a comment line to start a new line (just like regular markdown)
#' * Add `toc: true` to YAML frontmatter to create a table of contents with links like the one at the 
#' top of this page that links to h1, h2 & h3's indented like so:
#'     * h1
#'         * h2
#'             * h3
#' * Modify YAML to change syntax highlighting style (I'm using zenburn), author, title, theme, and all the good stuff
#' you're used to setting in Rmd documents.
#' * Sub-bullets like the ones above are created by a `#' *` with 4 spaces per level of indentation.
#' * Surround text with `*` to *italicize*  
#' * Surround text with `**` to **bold**  
#' * Surround text with `***` to ***italicize & bold***  
#' * Skip lines with `#' <br>`
#' * Keep comments in code, but hide from printing in report with `#' <!-- this text will not print in report -->`  
#' * Add hyperlinks:
#'     * [Rmarkdown cheatsheet](http://rmarkdown.rstudio.com/RMarkdownCheatSheet.pdf)
#'     * [Rmarkdown Reference Guide](http://rmarkdown.rstudio.com/RMarkdownReferenceGuide.pdf)
#'     * [Compiling R notebooks from R Scripts](http://rmarkdown.rstudio.com/r_notebook_format.html)
#' 

# comments without the extra tick show up like this.  And get included in code blocks
# loading mtcars data
data(mtcars)

#' ## Messing with data
 
library('knitr')

#' ### 3 ways to print an object
#' ...specifically a data.frame in this case.  Ordered from least to most pretty (in my opinion).

print(head(mtcars))
knitr::kable(head(mtcars))
#' including `#+ results='asis'` chunk option for formatting
#+ results='asis'
knitr::kable(head(mtcars))

#' ### Plotting
plot(mtcars$mpg, mtcars$disp, col=mtcars$cyl, pch=19)

#' We can change the chunk options we would use for a code block using `knitr` by using a comment that starts with `#+`.
#' For example, to change the plot size, we can specify `#+ fig.width=4, fig.height=4` before plotting.  
#' <br>
#' A new chunk is automatically generated (chunk settings reset) whenever we add document text with `#'` or change.
#' However, it is possible to specify global chunk options, if desired.
#' chunk options again with `#+`.  

#' `#+ fig.width=4, fig.height=4` <!-- simply for illustrative purposes in the document-->
#+ fig.width=4, fig.height=4
plot(mtcars$mpg, mtcars$disp, col=mtcars$cyl, pch=19)


#' Small plots often render with strange resolution and relative sizings of labels, axes, etc.  The `dpi` chunk option can be used 
#' to fix this.  Just be sure to adjust the fig.width and fig.height accordingly.
#' 
#' **Bad plot: ** `#+ fig.width=2, fig.height=2`
#+ fig.width=2, fig.height=2
hist(mtcars$mpg)

#' **Good plot: ** `#+ fig.width=4, fig.height=4, dpi=50`
#+ fig.width=4, fig.height=4, dpi=50
hist(mtcars$mpg)

#' Generate a series of plots from a loop
#+ fig.width=3, fig.height=3
for(i in 1:ncol(mtcars)) hist(mtcars[,i], breaks=40, xlab='', main=names(mtcars)[i])

#' ### Let's build a random forest model  
#' ... and explore some model output.  First here's a big chunk of text from the random forest documentation:  
#' <br>
#' **Random forest documentation:**  
#' randomForest implements Breiman's random forest algorithm (based on Breiman and Cutler's original Fortran code) 
#' for classification and regression. It can also be used in unsupervised mode for assessing proximities among data points.  
#' <br>
#' **Note:**  
#' The forest structure is slightly different between classification and regression. For details on how the trees are stored, see the help page for getTree.    
#' <br>
#' If xtest is given, prediction of the test set is done “in place” as the trees are grown. If ytest is also given, and do.trace is set to some positive integer, 
#' then for every do.trace trees, the test set error is printed. Results for the test set is returned in the test component of the resulting randomForest object. 
#' For classification, the votes component (for training or test set data) contain the votes the cases received for the classes. If norm.votes=TRUE, the fraction 
#' is given, which can be taken as predicted probabilities for the classes.  
#' <br>
#' For large data sets, especially those with large number of variables, calling randomForest via the formula interface is not advised: There may be too 
#' much overhead in handling the formula.  
#' <br>
#' The “local” (or casewise) variable importance is computed as follows: For classification, it is the increase in percent of times a case is OOB 
#' and misclassified when the variable is permuted. For regression, it is the average increase in squared OOB residuals when the variable is permuted.  


# OK. now let's actually use random.forest
library('randomForest')
mtcars$am <- as.factor(mtcars$am)
rf <- randomForest(am~., ntree=100, data=mtcars)

#' Here's the resulting confusion matrix on the training data.  Not very clear to a non-technical or non-forest savvy audience.
print(rf)

#' ### Dynamic comments
#' We can get fancy and actually dynamically generate some commentary around these results.  That is we can auto-fill parts of our document text
#' with objects from the R environment.  This could be useful with analyses that involve stochastic 
#' elements changing from run to run like random forest.  Or any analysis where results are subject to change.    
#' <br>  
#' `r rf$confusion[1,1]` cars are correctly classified as 0.  
#' `r rf$confusion[2,2]` cars are correctly classified as 1.  
#' `r rf$confusion[1,2]` cars are misclassified as 1.  
#' `r rf$confusion[2,1]` cars are misclassified as 0.  
#' 
#' These numbers were generated by wrapping the R expression to excute into the ticks like so: 
#' I don't know how to write this within a `#'` comment without evaluating it, so I'm documenting here as a
#' character string:  
#+ echo=F, eval=T
cat("#' `r rf$confusion[2,1]` cars are misclassified as 0. ")
#' <br>
#' 
#' ### Generate comments in a loop
#' 
#' This is useful if you want to generate lots of text without writing it manually.  

#' `#+ results='asis'`
#+ results='asis'
for (i in 1:10) {
  rf <- randomForest(am~., ntree=100, data=mtcars)
  cat("iteration ",i, ": ", rf$confusion[1,1], "cars are correctly classified as 0", "\n")
  cat('\n')
}

#' ## Toggle chunk settings globally by R variables
#' Much like we used R objects to dynamically generate text to print in the document (in the form of comments),
#' we can use R objects to dynamically specify chunk options.  

#' When we set `evaluateStuff` to `TRUE` or `FALSE`, the following 3 chunks will evaluate (or not) as we choose.
#' We can toggle them all with one variable, instead of manually changing the chunk settings with `#+ eval=T`
#' in the R script multiple times.  Simply
#' include the variable you want to execute in the chunk comments with ticks.   
#' Like so:  **#+ eval=\`evaluateStuff\`**

evaluateStuff <- T

#+ eval=`evaluateStuff`
cat('first thing to evaluate')
#+ eval=`evaluateStuff`
cat('second thing to evaluate')
#+ eval=`evaluateStuff`
cat('third thing to evaluate')

#' <br>
#' Now let's just print the code and not evaluate anything.
evaluateStuff <- F

#+ eval=`evaluateStuff`
cat('first thing to evaluate')
#+ eval=`evaluateStuff`
cat('second thing to evaluate')
#+ eval=`evaluateStuff`
cat('third thing to evaluate')

#' ## Converting R Script to HTML/PDF  
#' If you did everyhing right, above this is the easy part.  Simply render the script as desired with the `render` function from `rmarkdown`.  

#' `rmarkdown::render('/Users/you/Documents/yourscript.R')`

Render reports directly from R scripts was originally published by andrew brooks at andrew brooks on March 05, 2015.

Blogging with Jekyll and R Markdown using knitr

2015-01-25T00:00:00+00:00

use knitr

One way to blog using R and Jekyll is to copy and paste every code chunk, output and plot into a plain vanilla markdown file by hand. This is cumbersome, especially for plots which need to be saved as images and embedded. I tried this for my first few posts. It was clunky.

A more seamless solution is to use knitr to convert R Markdown files to Jekyll friendly markdown files. Knitr allows you to write your entire post in R Markdown utilizing all the features of traditional markdown while conveniently running your code, printing your output and displaying your charts. And it looks fresh.

Rmd ==> Jekyll markdown

Knitr is good at transforming R Markdown to HTML, PDF or even Word files. Jekyll is good at turning traditional/Jekyll markdown to HTML (your website). There might be an elegant way to shove the HTML generated by knitr into a jekyll site, but I haven’t found one. I found it much easier to convert R Markdown to its close cousin of markdown that is compatible with Jekyll. This conversion really just boils down to handling the R chunks: saving the plots as SVGs, formatting the code chunks with R syntax highlighting and printing the output for selected chunks.

A convenient feature provided in knitr is the ability to suppress code, messages, warnings, errors and output printed to the console for specific code chunks using the echo, message, warning, error and eval parameters respectively.

fuzzy middle

I tried a few approaches, but this one from chepec both worked the best and made the most sense to me. I had to make a couple adjustments to make it work for my setup, but they were small and included below. As I started generating more posts from R Markdown, I found a need to overwrite specific posts (rather than all or none). So I added a parameter overwriteOne which takes a string input and will overwrite any post that partially matches it (a simple grep ignoring case). However, my additions are trivial – all the credit goes to chepec.

how to use KnitPost

Configure the directories at the top of KnitPost to match the file system of your blog. I could have included these as parameters, but I figured I wouldn’t re-architect my blog very often… so I left them hard-coded.
Create an R Markdown post and save it as a .Rmd file in rmd.path. Be sure to include the proper YAML front matter for Jekyll. This tripped me up initially. I forgot to change the date from the knitr style date (“Month, day YYYY”) that auto-generates when you create a new .Rmd to a Jekyll style date (‘YYYY-MM-DD’).
Run KnitPost to publish your R Markdown file.

KnitPost <- function(site.path='/pathToYourBlog/', overwriteAll=F, overwriteOne=NULL) {
  if(!'package:knitr' %in% search()) library('knitr')
  
  ## Blog-specific directories.  This will depend on how you organize your blog.
  site.path <- site.path # directory of jekyll blog (including trailing slash)
  rmd.path <- paste0(site.path, "_Rmd") # directory where your Rmd-files reside (relative to base)
  fig.dir <- "assets/Rfig/" # directory to save figures
  posts.path <- paste0(site.path, "_posts/articles/") # directory for converted markdown files
  cache.path <- paste0(site.path, "_cache") # necessary for plots
  
  render_jekyll(highlight = "pygments")
  opts_knit$set(base.url = '/', base.dir = site.path)
  opts_chunk$set(fig.path=fig.dir, fig.width=8.5, fig.height=5.25, dev='svg', cache=F, 
                 warning=F, message=F, cache.path=cache.path, tidy=F)   
  

  setwd(rmd.path) # setwd to base
  
  # some logic to help us avoid overwriting already existing md files
  files.rmd <- data.frame(rmd = list.files(path = rmd.path,
                                full.names = T,
                                pattern = "\\.Rmd$",
                                ignore.case = T,
                                recursive = F), stringsAsFactors=F)
  files.rmd$corresponding.md.file <- paste0(posts.path, "/", basename(gsub(pattern = "\\.Rmd$", replacement = ".md", x = files.rmd$rmd)))
  files.rmd$corresponding.md.exists <- file.exists(files.rmd$corresponding.md.file)
  
  ## determining which posts to overwrite from parameters overwriteOne & overwriteAll
  files.rmd$md.overwriteAll <- overwriteAll
  if(is.null(overwriteOne)==F) files.rmd$md.overwriteAll[grep(overwriteOne, files.rmd[,'rmd'], ignore.case=T)] <- T
  files.rmd$md.render <- F
  for (i in 1:dim(files.rmd)[1]) {
    if (files.rmd$corresponding.md.exists[i] == F) {
      files.rmd$md.render[i] <- T
    }
    if ((files.rmd$corresponding.md.exists[i] == T) && (files.rmd$md.overwriteAll[i] == T)) {
      files.rmd$md.render[i] <- T
    }
  }
  
  # For each Rmd file, render markdown (contingent on the flags set above)
  for (i in 1:dim(files.rmd)[1]) {
    if (files.rmd$md.render[i] == T) {
      out.file <- knit(as.character(files.rmd$rmd[i]), 
                      output = as.character(files.rmd$corresponding.md.file[i]),
                      envir = parent.frame(), 
                      quiet = T)
      message(paste0("KnitPost(): ", basename(files.rmd$rmd[i])))
    }     
  }
  
}

Blogging with Jekyll and R Markdown using knitr was originally published by andrew brooks at andrew brooks on January 25, 2015.

Latent Dirichlet Allocation - under the hood

2015-01-17T00:00:00+00:00

LDA for mortals

I’ve been intrigued by LDA topic models for a few weeks now. I actually built one before I really knew what I was doing. That kind of frightens and excites me. On one hand, LDA provides rich output which is easy for the humanist researcher to interpret. On the other hand, shouldn’t I know what kind of machinery I’m operating that’s spitting out results? I would certainly like to know more than what’s provided in documentation of whatever software implementation I’m using (currently gensim), but I don’t necessarily need to be able to prove every last algorithm before I kick the tires and start building models.

I echo Ted Underwood’s observation that it can be difficult to develop a basic intuition of LDA from the hardcore computer science literature. Proving that it works, understanding how it works, and making it work, are very different things. My goal for this exercise was to bridge the gaps in my brain between the technical theoretical papers and the high level LDA articles out there by developing a rudimentary LDA model from scratch.

Generative, so what?

Most academic papers I encountered begin by describing LDA as a generative model. That is, a model that can randomly generate observable data (documents). Blei, Ng, & Jordan, 2003 outline this process in their seminal paper on the topic:

LDA assumes the following generative process for each document w in a corpus D:

Choose N ∼ Poisson(ξ).

Choose θ ∼ Dir(α).

For each of the N words w_n:
(a) Choose a topic z_n ∼ Multinomial(θ).
(b) Choose a word w_n from p(w_n | z_n ,β), a multinomial probability conditioned on the topic z_n.

I found this confusing at first. I was initially mixing up the generative and inference/training algorithms. They are different, very different. I already have my documents…why would I want to probabilistically generate new documents with my LDA model? Short answer, you probably don’t. However, the generative approach lets you do some cool things:

Assign probabilities to just about everything: words, documents and topics. Probabilistic Latent Semantic Indexing (pLSI), the precursor to LDA, pushed some boundaries in information retrieval, but fell short in providing a probabilistic model at the document level.
The ability to calculate probabilities (topic assignments) for previously unseen documents, which is not possible with pLSI.
Model updates. That is, you can continue training your model with new documents when they come in without repeatedly starting from scratch and re-training on the full corpus. Online learning should be supported in most LDA implementations.

There is even some new research (Zhai and Boyd-Graber, 2013) exploring breeds of LDA that allow for infinite vocabularies – assigning probabilities to new documents with words outside of the training vocabulary.

Statistical inference… how we actually train an LDA model

So the goal of LDA is to compute the 2 hidden parameters often commonly denoted as θ and φ, where θ represents topic-document distribution and φ represents the word-topic distribution.

However, when you do some fancy math, it becomes clear that the posterior distribution for these parameters is intractable. That is, we’re left with integrals that we cannot compute analytically in high dimensional space where numerical approximation gets hairy. How to handle this issue is the topic of a large swath of the LDA literature. It seems that most advocate one of two methods:

Variational inference seems to be the preferred method for getting fast and accurate results in most software implementations.
Markov chain Monte Carlo (sampling) methods have the benefit of being unbiased and easy understand. However, it seems MCMCs are handicapped by being computationally inefficient when working with large corpora.

Building LDA from scratch

Purpose

I have no intention for anyone (including myself) to use this code for anything beyond pedagogical purposes. There are countless ways to optimize the following approach. I simply wanted to put the theory into action as clearly as possible and dispel any myths in my head that LDA is black magic. I leave the implementation and optimization to the experts.

Getting started

Of all the academic papers I read trying to wrap my head around LDA, I found Streyver & Griffith’s Probabilistic Topic Models paper to be the most helpful for understanding implementation. Their 2004 paper, Finding Scientific Topics that everyone seems to cite provides a more thorough derivation of the Baysesian magic, but I found their subsequent paper most useful. They clearly walk through how to conduct inference using Collapsed Gibbs Sampling which I translated into R code.

Edwin Chen’s Introduction to Latent Dirichlet Allocation post provides an example of this process using Collapsed Gibbs Sampling in plain english which is a good place to start.

I did find some other homegrown R and Python implementations from Shuyo and Matt Hoffman – also great resources. Especially Shuyo’s code which I modeled my implementation after.

So let’s code it

I generated a very trivial corpus of 8 documents. To focus just on the LDA mechanics, I opted for the simplest of R objects: lists and matrices.

## Generate a corpus
rawdocs <- c('eat turkey on turkey day holiday',
          'i like to eat cake on holiday',
          'turkey trot race on thanksgiving holiday',
          'snail race the turtle',
          'time travel space race',
          'movie on thanksgiving',
          'movie at air and space museum is cool movie',
          'aspiring movie star')
docs <- strsplit(rawdocs, split=' ', perl=T) # generate a list of documents

## PARAMETERS
K <- 2 # number of topics
alpha <- 1 # hyperparameter. single value indicates symmetric dirichlet prior. higher=>scatters document clusters
eta <- .001 # hyperparameter
iterations <- 3 # iterations for collapsed gibbs sampling.  This should be a lot higher than 3 in practice.

print(docs)

## [[1]]
## [1] "eat"     "turkey"  "on"      "turkey"  "day"     "holiday"
## 
## [[2]]
## [1] "i"       "like"    "to"      "eat"     "cake"    "on"      "holiday"
## 
## [[3]]
## [1] "turkey"       "trot"         "race"         "on"          
## [5] "thanksgiving" "holiday"     
## 
## [[4]]
## [1] "snail"  "race"   "the"    "turtle"
## 
## [[5]]
## [1] "time"   "travel" "space"  "race"  
## 
## [[6]]
## [1] "movie"        "on"           "thanksgiving"
## 
## [[7]]
## [1] "movie"  "at"     "air"    "and"    "space"  "museum" "is"     "cool"  
## [9] "movie" 
## 
## [[8]]
## [1] "aspiring" "movie"    "star"

Abstracting the words away - now we have a numbers problem.

## Assign WordIDs to each unique word
vocab <- unique(unlist(docs))

## Replace words in documents with wordIDs
for(i in 1:length(docs)) docs[[i]] <- match(docs[[i]], vocab)

print(docs)

## [[1]]
## [1] 1 2 3 2 4 5
## 
## [[2]]
## [1] 6 7 8 1 9 3 5
## 
## [[3]]
## [1]  2 10 11  3 12  5
## 
## [[4]]
## [1] 13 11 14 15
## 
## [[5]]
## [1] 16 17 18 11
## 
## [[6]]
## [1] 19  3 12
## 
## [[7]]
## [1] 19 20 21 22 18 23 24 25 19
## 
## [[8]]
## [1] 26 19 27

Now we need to generate some basic count matrices. First we randomly assign topics to each word in each document. Then we create a word-topic count matrix.

## 1. Randomly assign topics to words in each doc.  2. Generate word-topic count matrix.
wt <- matrix(0, K, length(vocab)) # initialize word-topic count matrix
ta <- sapply(docs, function(x) rep(0, length(x))) # initialize topic assignment list
for(d in 1:length(docs)){ # for each document
  for(w in 1:length(docs[[d]])){ # for each token in document d
    ta[[d]][w] <- sample(1:K, 1) # randomly assign topic to token w.
    ti <- ta[[d]][w] # topic index
    wi <- docs[[d]][w] # wordID for token w
    wt[ti,wi] <- wt[ti,wi]+1 # update word-topic count matrix     
  }
}

print(wt) # Word-topic count matrix

##      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13]
## [1,]    2    2    0    0    1    1    0    0    1     1     2     1     0
## [2,]    0    1    4    1    2    0    1    1    0     0     1     1     1
##      [,14] [,15] [,16] [,17] [,18] [,19] [,20] [,21] [,22] [,23] [,24]
## [1,]     0     0     0     0     2     1     0     1     1     1     1
## [2,]     1     1     1     1     0     3     1     0     0     0     0
##      [,25] [,26] [,27]
## [1,]     1     0     0
## [2,]     0     1     1

print(ta) # Token-topic assignment list

## [[1]]
## [1] 1 1 2 1 2 2
## 
## [[2]]
## [1] 1 2 2 1 1 2 1
## 
## [[3]]
## [1] 2 1 1 2 2 2
## 
## [[4]]
## [1] 2 2 2 2
## 
## [[5]]
## [1] 2 2 1 1
## 
## [[6]]
## [1] 2 2 1
## 
## [[7]]
## [1] 1 2 1 1 1 1 1 1 2
## 
## [[8]]
## [1] 2 2 2

Now we generate a document-topic count matrix where the counts correspond to the number of tokens assigned to each topic for each document.

dt <- matrix(0, length(docs), K)
for(d in 1:length(docs)){ # for each document d
  for(t in 1:K){ # for each topic t
    dt[d,t] <- sum(ta[[d]]==t) # count tokens in document d assigned to topic t   
  }
}

print(dt) # document-topic count matrix

##      [,1] [,2]
## [1,]    3    3
## [2,]    4    3
## [3,]    2    4
## [4,]    0    4
## [5,]    2    2
## [6,]    1    2
## [7,]    7    2
## [8,]    0    3

I’m beginning to mix notation from the code and theory. Oops. Anything in italics refers to theory. Anything in an inline code block refers to my iterators and objects in R code.

If following along with Streyvers and Griffiths, wt corresponds to the C^WT matrix and dt corresponds to the C^DT matrix.

Now we get to the fun part where LDA starts learning topics. We will iteratively update the random (bad) topic assignments we populated into ta one token at a time. This is where I had to brush up on my Bayesian statistics.

Rather than directly estimating our parameters of interest (φ and θ), we can directly esimate the posterior over z which represents topic assignments for each token. This is p_z, the full-conditional distribution for z which represents the probability that token w belongs to topic t.

The left side of p_z ((wt[,wid] + eta) / denom_b) represents the probability of word w under topic t. After many tokens of a word have been assigned to a particular topic, LDA increasingly wants to assign new occurrences of this token to that topic.

The right side of p_z ((dt[d,] + alpha) / denom_a) represents the probablity of topic t in document d. If many tokens in a document have been assigned to a particular topic, LDA wants to assign tokens to to that topic at each iteration of Gibbs Sampling.

Example: So if 90% of the tokens in a document belong to topic 1 at the start of an iteration of Gibbs Sampling AND those same tokens in other documents are assigned to topic 1, it’s a no-brainer for LDA. Topic 1 it is.

However, if those 90% of tokens that belonged to topic 1 in our document all belong to topic 2 in the other documents, then it’s a toss-up and LDA is forced to spread the wealth across the two topics.

Defining some notation…

p_z= P(z_i = j | z_-i, w_i, d_i)
where
P(z_i = j) is the probability that token i is assigned to topic j
z_-i represents topic assignments of all other tokens
w_i is the word index of token i (1…length(vocab))
d_i is the document index of token i (1…length(docs))

This is really the only “magic” part of the whole LDA learning process far. To dispel the magic and truly understand why this works read Griffiths and Streyvers – they walk through the derivation of the full-conditional distribution (p_z) much more elegantly than I could.

for(i in 1:iterations){ # for each pass through the corpus
  for(d in 1:length(docs)){ # for each document
    for(w in 1:length(docs[[d]])){ # for each token 
      
      t0 <- ta[[d]][w] # initial topic assignment to token w
      wid <- docs[[d]][w] # wordID of token w
      
      dt[d,t0] <- dt[d,t0]-1 # we don't want to include token w in our document-topic count matrix when sampling for token w
      wt[t0,wid] <- wt[t0,wid]-1 # we don't want to include token w in our word-topic count matrix when sampling for token w
      
      ## UPDATE TOPIC ASSIGNMENT FOR EACH WORD -- COLLAPSED GIBBS SAMPLING MAGIC.  Where the magic happens.
      denom_a <- sum(dt[d,]) + K * alpha # number of tokens in document + number topics * alpha
      denom_b <- rowSums(wt) + length(vocab) * eta # number of tokens in each topic + # of words in vocab * eta
      p_z <- (wt[,wid] + eta) / denom_b * (dt[d,] + alpha) / denom_a # calculating probability word belongs to each topic
      t1 <- sample(1:K, 1, prob=p_z/sum(p_z)) # draw topic for word n from multinomial using probabilities calculated above
      
      ta[[d]][w] <- t1 # update topic assignment list with newly sampled topic for token w.
      dt[d,t1] <- dt[d,t1]+1 # re-increment document-topic matrix with new topic assignment for token w.
      wt[t1,wid] <- wt[t1,wid]+1 #re-increment word-topic matrix with new topic assignment for token w.
    
      if(t0!=t1) print(paste0('doc:', d, ' token:' ,w, ' topic:',t0,'=>',t1)) # examine when topic assignments change
    }
  }
}

## [1] "doc:1 token:2 topic:1=>2"
## [1] "doc:1 token:4 topic:1=>2"
## [1] "doc:1 token:5 topic:2=>1"
## [1] "doc:1 token:6 topic:2=>1"
## [1] "doc:2 token:1 topic:1=>2"
## [1] "doc:2 token:3 topic:2=>1"
## [1] "doc:3 token:2 topic:1=>2"
## [1] "doc:3 token:3 topic:1=>2"
## [1] "doc:3 token:5 topic:2=>1"
## [1] "doc:3 token:6 topic:2=>1"
## [1] "doc:5 token:1 topic:2=>1"
## [1] "doc:5 token:2 topic:2=>1"
## [1] "doc:5 token:4 topic:1=>2"
## [1] "doc:7 token:1 topic:1=>2"
## [1] "doc:7 token:2 topic:2=>1"
## [1] "doc:7 token:6 topic:1=>2"
## [1] "doc:7 token:7 topic:1=>2"
## [1] "doc:1 token:5 topic:1=>2"
## [1] "doc:2 token:2 topic:2=>1"
## [1] "doc:2 token:3 topic:1=>2"
## [1] "doc:2 token:5 topic:1=>2"
## [1] "doc:7 token:3 topic:1=>2"
## [1] "doc:7 token:4 topic:1=>2"
## [1] "doc:7 token:8 topic:1=>2"
## [1] "doc:8 token:1 topic:2=>1"
## [1] "doc:8 token:3 topic:2=>1"
## [1] "doc:1 token:5 topic:2=>1"
## [1] "doc:2 token:2 topic:1=>2"
## [1] "doc:2 token:3 topic:2=>1"
## [1] "doc:2 token:5 topic:2=>1"
## [1] "doc:4 token:4 topic:2=>1"
## [1] "doc:5 token:1 topic:1=>2"
## [1] "doc:5 token:2 topic:1=>2"
## [1] "doc:7 token:2 topic:1=>2"

The printed output above is a window into the learning process. We printed a line each time a token got reassigned to a new topic. Remember we only made 3 passes (iterations <- 3) through the corpus, so our topic assignments are likely still pretty terrible. In practice, with many more iterations, these re-assignments would eventually stabilize.

So now we’ve finished learning topics. We just have to analyze them. theta normalizes the counts in the document-topic count matrix dt to compute the probability that a document belongs to each topic. theta is technically the posterior mean of the parameter θ.

theta <- (dt+alpha) / rowSums(dt+alpha) # topic probabilities per document
print(theta)

##        [,1]   [,2]
## [1,] 0.5000 0.5000
## [2,] 0.5556 0.4444
## [3,] 0.3750 0.6250
## [4,] 0.3333 0.6667
## [5,] 0.3333 0.6667
## [6,] 0.4000 0.6000
## [7,] 0.1818 0.8182
## [8,] 0.6000 0.4000

phi normalizes the word-topic count matrix wt to compute the probabilities of words given topics. phi is technically the posterior mean of the parameter φ.

phi <- (wt + eta) / (rowSums(wt+eta)) # topic probabilities per word
colnames(phi) <- vocab
print(phi)

##           eat    turkey        on      day  holiday         i      like
## [1,] 0.133160 6.655e-05 6.655e-05 0.066613 0.199707 6.655e-05 6.655e-05
## [2,] 0.000037 1.110e-01 1.480e-01 0.000037 0.000037 3.704e-02 3.704e-02
##            to     cake      trot      race thanksgiving     snail
## [1,] 0.066613 0.066613 6.655e-05 6.655e-05     0.133160 6.655e-05
## [2,] 0.000037 0.000037 3.704e-02 1.110e-01     0.000037 3.704e-02
##            the   turtle      time    travel    space     movie        at
## [1,] 6.655e-05 0.066613 6.655e-05 6.655e-05 0.133160 6.655e-05 6.655e-05
## [2,] 3.704e-02 0.000037 3.704e-02 3.704e-02 0.000037 1.480e-01 3.704e-02
##            air       and    museum        is      cool aspiring     star
## [1,] 6.655e-05 6.655e-05 6.655e-05 6.655e-05 6.655e-05 0.066613 0.066613
## [2,] 3.704e-02 3.704e-02 3.704e-02 3.704e-02 3.704e-02 0.000037 0.000037

Latent Dirichlet Allocation - under the hood was originally published by andrew brooks at andrew brooks on January 17, 2015.

New York Times Article Search API to MongoDB

2015-01-06T00:00:00+00:00

Motivation
Accessing NYT API
Extracting and parsing the article body text
Writing to MongoDB
Pipeline
Results

Motivation

I’ve learned a little about a lot of different corners of the text mining and NLP world over the last few years… which sometimes makes me feel like I know nothing for certain. I’ve done a decent amount web scraping, processing HTML and parsing text recently, but never a full blown text mining project. I decided to start with some topic modeling using Latent Dirichlet Allocation and document clustering. Unsupervised learning techniques requiring minimal upfront work beyond the text pre-processing seemed like a good (and interesting) place to get started.

After surveying APIs for a few news sources, the New York Times seemed to be the most robust. I wanted the flexibility to acquire new documents in the future for testing models and from far enough in the past to build a hefty corpus. I also wanted to have some fun and scrape the documents myself, experiment with a NoSQL database pipeline and process the HTML from the rawest form.

Accessing NYT API

API Documentation and keys

The New York Times API is well documented and user-friendly. I didn’t experiment too much with targeted querying since I was pulling all articles over a period of time. However the q and fq parameters seem to provide a lot of functionality for filtered searches.

Requesting an a key for the Article Search API was easy and instantaneous. I saved my key as a global parameter using R options. I’m using sample-key here simply for illustrative purposes, but I found that this key (provided by default in the NYT API Console) actually works.

options(nyt_as_key = 'sample-key')  ## copy and paste your API key here.
options()$nyt_as_key

## [1] "sample-key"

The NYT API Console is also a nifty way to kick the tires and understand the parameters at your disposal for constructing queries. It’s basically a GUI around the API which gives you everything you need on one page to quickly formulate a query, submit it and inspect the response. It also allowed me to quickly determine that this is a well developed and documented API worthy of pursuing further. I wish all APIs were like this… sigh.

Generate URL

This is the easy part. I did find an R package rtimes for accessing the API which worked for the few queries I tried. However, I had already started building my own pipeline, so I stuck with it.

The NYT API returns only 10 articles per request. Which 10 are dictated by the page parameter. page=0 returns articles 1-10, page=1 returns articles 11-20, etc. The tricky part is knowing how many pages to iterate through. On my first pass, I failed to find a way to figure out the total number of articles matching my query to tell me how many pages and API requests I would need. So instead, I simply started my requests with page=0 and incrementally added 1 to page until the response stopped yielding articles.

However, after I already maxed out my database, I found a way to do this using the facet_field and facet_filter parameters. These can be used to count the number of articles matching a filtered query by simply adding &facet_field=source&facet_filter=true to your query URL.

And you get something like this:

{  
   "facets":{  
      "source":{  
         "terms":[  
            {  
               "term":"The New York Times",
               "count":159
            },
            {  
               "term":"International Herald Tribune",
               "count":7
            },
            {  
               "term":"",
               "count":1
            }
         ]
      }
   },
   "status":"OK",
   "copyright":"Copyright (c) 2013 The New York Times Company.  All Rights Reserved."
}

Here’s an example using R and httr to get the same result¹. These counts almost perfectly match the counts I’ve calculated from my collection for the New York Times and International Herald Tribune. However, I’ve collected a lot of Reuters and AP articles from the Article Search API that interestingly don’t appear here.

library('httr')
resp <- GET('http://api.nytimes.com/svc/search/v2/articlesearch.json?facet_field=source&facet_filter=true&begin_date=20130105&end_date=20130105&api-key=sample-key')
print(content(resp, 'parsed')$response$facets)

## $source
## $source$terms
## $source$terms[[1]]
## $source$terms[[1]]$term
## [1] "The New York Times"
## 
## $source$terms[[1]]$count
## [1] 159
## 
## 
## $source$terms[[2]]
## $source$terms[[2]]$term
## [1] "International Herald Tribune"
## 
## $source$terms[[2]]$count
## [1] 7
## 
## 
## $source$terms[[3]]
## $source$terms[[3]]$term
## [1] ""
## 
## $source$terms[[3]]$count
## [1] 1

Another rub is that “pagination beyond page 100 is not allowed at this time.” So if you’re extracting large quantities of articles, best do it chunks. I chunked my queries into single days – usually returning about 700-800 articles. Without using any facets or query terms, I ran the risk of only collecting 1000 articles for a day when there were more than 1000. This occurred 12 days out of 120 for me.

makeURL is a pretty trivial function that generates the URL to make the GET request from a collection of NYT API parameters. However, I found encapsulating this step in a function which gets called in subsequent functions kept things clean.

makeURL <- function(q=NULL, fq=NULL, begin_date=NULL, end_date=NULL, key=getOption("nyt_as_key"), page=0, 
                    sort=NULL, fl=NULL, hl=NULL, facet_field=NULL, facet_filter=NULL){
  arglist <- list(q=q, fq=fq, begin_date=begin_date, end_date=end_date, 'api-key'=key, page=page,
                  sort=sort, fl=fl, hl=hl, facet_field=facet_field, facet_filter=facet_filter)
  url <- 'http://api.nytimes.com/svc/search/v2/articlesearch.json?'
  for(i in 1:length(arglist)){
    if(is.null(unlist(arglist[i]))==F){
      url <- paste0(url, '&', names(arglist[i]), '=', arglist[i])
    }
  }
  return(url)
}

## example: generate query which will return all articles for one day.
library('httr')
url <- makeURL(begin_date='20130101', end_date='20130102', key='sample-key', page=100)
print(url)

## [1] "http://api.nytimes.com/svc/search/v2/articlesearch.json?&begin_date=20130101&end_date=20130102&api-key=sample-key&page=100"

Scrape NYT metadata

Here in getMeta we’re actually collecting the NYT article metadata and structuring it as a list object. Like most things, the meat of it is pretty simple. The rest is parsing, exception and error handling… which I find are worth it with web scraping, especially if you’re running a job overnight and want it to work by the time you wake up.

getMeta:

DOES iterate through pages until no new articles are returned.
DOES sleep for sleep seconds after each request. I had good luck with sleep=0.1.
DOES re-attempt failed requests tryn times. Failed requests were almost always successful after the second try, so I set tryn=3 just to be safe.
DOES NOT cache responses to disc. Everything is saved in memory and returned as a list in R memory after the function is complete. I opted not to bother with caching with this step as it only takes ~20 seconds to run through 100 API calls (the max per query).

library('httr')

getMeta <- function(url, pages=Inf, sleep=0.1, tryn=3) {
  art <- list()
  i <- 1
  e <- seq(-tryn, -tryn/2, length.out=tryn) # initialize list of failed pages with arbitrary negative numbers
  while(i<=pages){
    if(length(unique(e[(length(e)-(tryn-1)):length(e)]))==1) i <- i+1 ## attempt tryn times before moving on
    tryget <- try({
      urlp <- gsub('page=\\d+', paste0('page=', i), url) ## get the next page
      p <- GET(urlp) ## actually make GET request
      pt <- content(p, 'parsed') ## retrieve contents of response formatted in a nested R list
      if(length(pt$response$docs)>0) art <- append(art, pt$response$docs) ## add articles to list
      else {print(paste0(i, ' pages collected')); break}
      if(i %% 10 ==0) print(paste0(i, ' pages of metadata collected'))
      i <- i+1
    })
    
    ## if there was  a fail...
    if(class(tryget)=='try-error') {
      print(paste0(i, ' error - metadata page not scraped'))
      e <- c(e, i) ## add page i to failed list
      e <- e[(length(e)-(tryn-1)):length(e)] ## keep the failed list to just tryn elements
      Sys.sleep(0.5) ## probably scraping too fast -- slowing down
    }
    Sys.sleep(sleep)
  }
  return(art)
}

## example: collect metadata for 3 pages (30 articles)
meta <- getMeta(url, pages=3)
str(meta[[1]]) ## look at just one article

## List of 19
##  $ web_url         : chr "http://thecaucus.blogs.nytimes.com/2013/01/02/obama-returns-to-hawaii-to-continue-vacation/"
##  $ snippet         : chr "President Obama eased back into the relaxing groove of his Christmas in Hawaii vacation after a brief hiatus to wrangle with Co"| __truncated__
##  $ lead_paragraph  : NULL
##  $ abstract        : chr "President Obama eased back into the relaxing groove of his Christmas in Hawaii vacation after a brief hiatus to wrangle with Co"| __truncated__
##  $ print_page      : NULL
##  $ blog            : list()
##  $ source          : chr "The New York Times"
##  $ multimedia      : list()
##  $ headline        :List of 2
##   ..$ main  : chr "Obama Returns to Hawaii to Continue Vacation"
##   ..$ kicker: chr "The Caucus"
##  $ keywords        :List of 3
##   ..$ :List of 3
##   .. ..$ rank : chr "1"
##   .. ..$ name : chr "persons"
##   .. ..$ value: chr "Obama, Barack"
##   ..$ :List of 3
##   .. ..$ rank : chr "1"
##   .. ..$ name : chr "glocations"
##   .. ..$ value: chr "Hawaii"
##   ..$ :List of 3
##   .. ..$ rank : chr "1"
##   .. ..$ name : chr "subject"
##   .. ..$ value: chr "United States Politics and Government"
##  $ pub_date        : chr "2013-01-02T18:37:03Z"
##  $ document_type   : chr "blogpost"
##  $ news_desk       : NULL
##  $ section_name    : chr "U.S."
##  $ subsection_name : NULL
##  $ byline          :List of 2
##   ..$ person  :List of 1
##   .. ..$ :List of 6
##   .. .. ..$ firstname   : chr "Jeremy"
##   .. .. ..$ middlename  : chr "W."
##   .. .. ..$ lastname    : chr "PETERS"
##   .. .. ..$ rank        : int 1
##   .. .. ..$ role        : chr "reported"
##   .. .. ..$ organization: chr ""
##   ..$ original: chr "By JEREMY W. PETERS"
##  $ type_of_material: chr "Blog"
##  $ _id             : chr "50e4c5c700315214fbb821e4"
##  $ word_count      : int 366

Extracting and parsing the article body text

So now we have a lot of information about NYT articles including URL, abstract, headline, publication date, section, author, type of material, etc. However, notably absent is the article text itself. The process for collecting article body text is much more manual. We’ve used the API road as much as we can, but now we have to go off-road a little bit and collect each article’s body text from its individual URL provided from the metadata field web_url.

After a while of trial-and-error and guessing and checking, I developed a basic utility function parseArticleBody to strip just the article body text from the raw HTML. At least one of three simple XPath queries seemed to work reasonably well for the normal plain vanilla articles. Most of the video and photography pages contain little to no text, so these often come up empty. Some of the more modern pages like this one which utilize multimedia and spread content over several pages usually fail too.

However, there’s enough NYT articles in the world to sink a small ship², so I’m more concerned with precision than recall – I’m willing to let some articles slip through the cracks if it boosts the quality of text for the articles I am able to extract it for. In most cases the gains from nailing the XPath query were marginal as the most common cause for missing article body text was out-of-date URLs provided by the NYT API.

library('XML')
library('httr')

parseArticleBody <- function(artHTML) {
  xpath2try <- c('//div[@class="articleBody"]//p',
                 '//p[@class="story-body-text story-content"]',
                 '//p[@class="story-body-text"]'
                 )
  for(xp in xpath2try) {
    bodyi <- paste(xpathSApply(htmlParse(artHTML), xp, xmlValue), collapse='')
    if(nchar(bodyi)>0) break
  }
  return(bodyi)
}

## Example: extract article text for one article
p <- GET('http://www.nytimes.com/2013/01/31/garden/faketv-can-fool-intruders.html')
html <- content(p, 'text')
artBody <- parseArticleBody(html)
print(artBody)

## [1] "\nYou might not be home every evening watching television, but no burglar needs to know that. The coruscating light of a television set — signature feature of an occupied home — might be enough to scare prowlers off.        \nThat’s the idea behind FakeTV ($35), a security gadget that replicates a television’s flickering light in all of its dynamic variety, from kinetic commercials to subdued news reports.        \nThe lightweight wedge, smaller than a slice of cake, is the brainchild of Blaine Readler, an engineer, who intentionally left his TV on when going out one night. The fake version, which uses bright LEDs, saves wear and tear on a real TV, as well as the cost of electricity. Energy-wise, it’s equivalent to a 2-watt night light, said Rein Teder, president of the manufacturer, Hydreon Corporation.        \nAnd unlike most TVs today, which turn on and off with buttons or remotes, FakeTV can be put on a timer. Information: faketv.com or (877) 532-5388.         "

Writing to MongoDB

Choosing MongoDB

Github was a logical place to store code for this project – my blog is already hosted there and integrates well with my workflow allowing me to work from my home, office or villa on a remote island³. Where to store the data was less obvious. I wanted something more robust than a directory full of text files… some sort of NoSQL database that I could access from Python and R. I also wanted to make the data accessible via the web. And I wanted it all for free.

mongolab (MongoDB’s cloud database-as-a-service) fit the bill, for 500MB at least. I don’t have much experience with NoSQL databases and haven’t worked with Mongo in the past, but I had a lot of fun learning and working with Mongo. My grasp of the Mongo querying language is still tenuous, but I was able make it do everything I needed it to do.

I found the web interface intuitive and useful for testing queries or other Mongo command line operations when I was getting started. From here you can also create database users with read-only or full access – useful for sharing your work with the world while holding the keys to the kingdom to yourself.

Working with MongoDB in Python and R

I used the RMongo R package and the pymongo Python package for interacting with the database. No complaints on either. I like how the usage inside Python and R using these packages seems to mirror pretty closely usage at the command line.

When the cloud can bring you down

So high, yet so low. I did experience a period of down-time where I was unable to access my database for an hour or so. Although mongolab does a decent job of documenting issues on their status page, the list doesn’t appear to be short… As is life sometimes, I suppose, when you’re on the free-tier of a cloud service.

Pulling it all together

getArticles does most of the heaviest lifting – extracts, parses and adds the article body text to the meta object and inserts to MongoDB after scraping each article.

getArticles:

DOES use the list of metadata as a starting point (the meta argument). Then scrapes, parses and adds the body as an attribute (text string) to the article’s metadata. Only adds to the meta object.
DOES cache – writes to MongoDB after extracting each article. I didn’t bump into any rate-limits from MongoDB and I had to wait a decisecond or two after each article anyway, so insert speed was not an issue for me. Had all the articles been pulled into memory first, I imagine a bulk insert would be more efficient.
DOES convert the R list of metadata for each article to JSON on the fly using the toJSON function which is then conveniently inserted to MongoDB as-is. MongoDB likes JSON.
DOES re-attempt failed article extraction, parsing and MongoDB inserts – If there was an error in one of these steps, I tried another 2 times before moving on to the next article.
DOES include parameters:
- meta is created from the function getMeta.
- n is the number of articles to scrape.
- overwrite decides whether to start scraping at the beginning or the first article that does not have a body attribute.
- sleep is the time in seconds to pause after each article. Defaults to 0.1 seconds.
- mongo is list of credentials used to write to the database. To return results in memory and not write to database, specify mongo=NULL.

library('XML')
library('RMongo')
library('rjson')

getArticles <- function(meta, n=Inf, overwrite=F, sleep=0.1, mongo=list(dbName, collection, host='127.0.0.1', username, password, ...)) {
  metaArt <- meta
  if(is.null(mongo$dbName)==F){
    con <- mongoDbConnect(mongo$dbName, mongo$host)
    try(authenticated <-dbAuthenticate(con, username=mongo$username, password=mongo$password))
  }
  if(overwrite==T) {artIndex <- 1:(min(n, length(meta)))
  } else { 
    ii <-  which(sapply(meta, function(x) is.null(x[['bodyHTML']]))) ## get index of articles that have not been scraped yet
    artIndex <- ii[1:min(n,length(ii))] ## take first n articles
  }
  e <- c(-3,-2,-1) # initialize failed list of articles with arbitrary negative numbers
  i<-1
  while(i<=length(artIndex)){
    if(length(unique(e[(length(e)-2):length(e)]))==1) i <- i+1 ## if we tried and failed 3 times, move on
    tryget <- try({
      if(overwrite==T | (is.null(meta[[i]]$body)==T & overwrite==F)) {
        p <- GET(meta[[i]]$web_url)
        html <- content(p, 'text') ## metaArt[[i]]$bodyHTML <- content(p, 'text')
        metaArt[[i]]$body <- parseArticleBody(html)
        if(is.null(mongo$dbName)==F) dbInsertDocument(con, mongo$collection, toJSON(metaArt[[i]]))
        if(i %% 10==0) print(paste0(i, ' articles scraped'))
        i<-i+1
      }
    })
    if(class(tryget)=='try-error') {
      print(paste0(i, ' error - article not scraped'))
      e <- c(e, i)
      e <- e[(length(e)-2):length(e)]
      Sys.sleep(0.5) ## probably scraping too fast -- slowing down
    }
    Sys.sleep(sleep)
  }
  return(metaArt)
}

## Example: get article bodies for first 10 articles.  
art <- getArticles(meta, n=3, mongo=NULL) ## scrape article body text for first 3 articles.  Do not write to MongoDB
length(art) ## we only add to meta.  Still returns articles for which we did not collect body text.

## [1] 30

str(art[[1]]) ## we now have a "body" attribute.

## List of 20
##  $ web_url         : chr "http://thecaucus.blogs.nytimes.com/2013/01/02/obama-returns-to-hawaii-to-continue-vacation/"
##  $ snippet         : chr "President Obama eased back into the relaxing groove of his Christmas in Hawaii vacation after a brief hiatus to wrangle with Co"| __truncated__
##  $ lead_paragraph  : NULL
##  $ abstract        : chr "President Obama eased back into the relaxing groove of his Christmas in Hawaii vacation after a brief hiatus to wrangle with Co"| __truncated__
##  $ print_page      : NULL
##  $ blog            : list()
##  $ source          : chr "The New York Times"
##  $ multimedia      : list()
##  $ headline        :List of 2
##   ..$ main  : chr "Obama Returns to Hawaii to Continue Vacation"
##   ..$ kicker: chr "The Caucus"
##  $ keywords        :List of 3
##   ..$ :List of 3
##   .. ..$ rank : chr "1"
##   .. ..$ name : chr "persons"
##   .. ..$ value: chr "Obama, Barack"
##   ..$ :List of 3
##   .. ..$ rank : chr "1"
##   .. ..$ name : chr "glocations"
##   .. ..$ value: chr "Hawaii"
##   ..$ :List of 3
##   .. ..$ rank : chr "1"
##   .. ..$ name : chr "subject"
##   .. ..$ value: chr "United States Politics and Government"
##  $ pub_date        : chr "2013-01-02T18:37:03Z"
##  $ document_type   : chr "blogpost"
##  $ news_desk       : NULL
##  $ section_name    : chr "U.S."
##  $ subsection_name : NULL
##  $ byline          :List of 2
##   ..$ person  :List of 1
##   .. ..$ :List of 6
##   .. .. ..$ firstname   : chr "Jeremy"
##   .. .. ..$ middlename  : chr "W."
##   .. .. ..$ lastname    : chr "PETERS"
##   .. .. ..$ rank        : int 1
##   .. .. ..$ role        : chr "reported"
##   .. .. ..$ organization: chr ""
##   ..$ original: chr "By JEREMY W. PETERS"
##  $ type_of_material: chr "Blog"
##  $ _id             : chr "50e4c5c700315214fbb821e4"
##  $ word_count      : int 366
##  $ body            : chr "\tHONOLULU – President Obama eased back into the relaxing groove of his Christmas in Hawaii vacation on Wednesday after a brief"| __truncated__

print(art[[1]]['body']) ## full body text of the article.

## $body
## [1] "\tHONOLULU – President Obama eased back into the relaxing groove of his Christmas in Hawaii vacation on Wednesday after a brief hiatus to wrangle with Congress over a deal to avert a fiscal crisis.\tIn most respects, Wednesday on the island of Oahu was 5,000 miles away and a world apart from the political brinksmanship of the nation’s capital. The morning included a trip to the local Marine Corps base, where the president went to the gym.\tAnd after speaking with Gov. Andrew M. Cuomo of New York and Gov. Chris Christie of New Jersey about the stalled efforts to provide tens of billions of dollars to the states to aid their recovery from Hurricane Sandy, Mr. Obama hit the golf course. \tThe White House announced Mr. Obama’s golf partners, as is standard protocol, and they once again included a friend who always creates a bit of a stir whenever he is spotted with the president. Bobby Titcomb, an old friend of the president’s from their days at the Punahou School here, was arrested in 2011 on a charge of soliciting prostitution after being swept up in an undercover sting. He later entered a plea of no contest. \tThe president’s group also included Marty Nesbitt, a Chicago friend. Mr. Nesbitt was an early backer of Mr. Obama, and he later served as treasurer for his 2008 presidential campaign. \tThe third in their foursome was Allison Davis, a Chicago developer and lawyer who was one of the founders of the firm the president once worked for and who was another financial backer of his early political career. \tThe White House also released a new taped message from the president on Wednesday in which he outlined his priorities for 2013 – listing them in order as winding down the war in Afghanistan, reforming immigration and gun control – and cautioned that many of the issues that made the tax-and-spending deal so difficult remain unresolved.\t“Obviously there’s still more to do when it comes to reducing our debt,” he said. “And I’m willing to do more as long as it does it in a balanced way that doesn’t put all the burden on seniors or students or middle-class families.”"

Pipeline

So putting it all together, the pipeline looks like this:

## dependencies
library('rjson')
library('httr')
library('RMongo')
library('XML')

## parameters
days <- gsub('-', '', seq(as.Date('2013-01-01'), Sys.Date()-1, by=1))
mongoCreds <- list(dbName='nyt', collection='articles1', host='ds063240.mongolab.com:63240', username='myUsername', password='myPassword')
key <- 'myKey' # replace with your personal key

## Letting it rip ... 
all <- list() ## persist all results in memory in addition to MongoDB... just in case.
for(d in days) {
  url <- makeURL(begin_date=d, end_date=d, key=key) # generate URL
  meta <- getMeta(url, pages=Inf, sleep=0.1) # extract metadata from NYT API
  artxt <- getArticles(meta, n=Inf, sleep=0.1, overwrite=T, mongo=mongoCreds) # extract article text and write to MongoDB
  print(paste0('day ',  d, ' complete at ', Sys.time()))
  all <- append(all, artxt) # persist results
}

Results

It took me about 1 full day to max out the 500MB in my free mongolab database. I ended up with 85,000 articles covering roughly 4 months of NYT articles. Since I’m specifically interested in text mining the article text, my next step will likely be to delete the documents where the article body could not be extracted.

A lot of the URLs provided from the NYT API were stale – only ~35% of URLs contained a real article with text. It wasn’t until I was doing some preliminary analysis poking around on the full corpus that I realized links to historic AP and Reuters articles (of which there are many) were included and almost universally not fruitful. These could easily be filtered out using the fq parameter in the NYT API: fq=source:("The New York Times").

## Crosstab of article source vs. whether article was successfully scraped 

                        **body scraped successfully**
  **Source**                   FALSE  TRUE
  The New York Times            6686 34322
  AP                           28944     8
  Reuters                      24001     0
  International Herald Tribune     9  1221
                                1007     4
  du_recipe                      273     0
  The Broadway Channel             8     0
  CNBC                             6     0
  ADAM                             1     0

I decided to use gensim and nltk in Python for the text pre-processing and topic modeling. More to come on that analysis and visualization.

Footnotes

Note the use of the content function from the httr package with as='parsed' is not recommended by the authors for use in other R packages or production systems. It’s provided as a convenience function… which I found, well, convenient… so I used it. ↩
In fact there are over 1000 articles alone matching the query parameters “ship+sink” ↩
Still working on the villa … ↩

New York Times Article Search API to MongoDB was originally published by andrew brooks at andrew brooks on January 06, 2015.

Scraping with Selenium

2014-12-11T00:00:00+00:00

If you’ve ever…

felt like you’re playing Simon Says with mouse clicks when repeatedly extracting data in chunks from a front-end interface to a database on the web, well, you probably are. There’s probably a better solution – Selenium.

ever used XML or httr in R or urllib2 in Python, you’ve probably encountered the situation where the source code you’ve scraped for a website doesn’t contain all the information you see in your browser. Selenium can probably help.

How it works

Selenium is a web automation tool. While not developed specifically for web scraping, Selenium does it pretty dang well. Selenium literally “drives” your browser, so it can see anything you see when you right click and inspect element in Chrome or Firefox. This vastly widens the universe of content that can be extracted from automation, but can be slow as all content must be rendered in the browser.

There are headless (invisible browsers with no GUI) such as phantomjs that speed some of this up. That said, I’ve found that Selenium works best for targeted extraction where the user knows exactly what they want.

Example

I set out to collect tickers for all mutual funds in the asset allocation fund type. Fidelity provides a list of all these funds here. 1,586 funds as of today in 80 conveniently paginated URLs. Each URL ends in &pgNo=5 to indicate you want page 5 (or whatever number between 1 and 80).

In my browser, when I hover my mouse over one of the fund names in the table, I see the 5 character ticker I’m looking for. I also see the tickers directly on the webpage when I click the link to each fund. Here for example, where it says PSLDX in the top left. However, if possible I’d like to scrape the tickers from the table rather than the individual fund pages. This would mean 80 pages to scrape rather than 1,586.

Take 1: traditional http request

When possible, it makes sense to use the simple traditional methods. So I first tried to extract these tickers with the popular httr R package by making standard http requests.

library('httr')
url <- 'https://www.fidelity.com/fund-screener/evaluator.shtml#!&ft=BAL_all&ntf=N&expand=%24FundType&rsk=5'
page <- GET(url)

Success…

print(http_status(page))

## $category
## [1] "success"
## 
## $message
## [1] "success: (200) OK"

But did our http request return the information we want?

page_text <- content(page, as='text')

Nope – can’t find the tickers (one of them anyway).

grepl('GMMAX', page_text, ignore.case=T)

## [1] FALSE

Nope – can’t even find the fund name that I see in the table from the webpage in my browser.

grepl('Aberdeen', page_text, ignore.case=T)

## [1] FALSE

It appears that the content of interest is being generated dynamically with Javascript and Ajax on this webpage. So the raw HTML of this page doesn’t help us much.

My plan B was to grab the url for each fund from the table, navigate to that fund’s page, and extract the ticker from there. However these links weren’t in our http response. I noticed that the URLs for each fund followed a simple consistent structure.
https://fundresearch.fidelity.com/mutual-funds/summary/72201F433 for example. I thought maybe I could find 72201F433 which looks like some sort of fund ID in a list with all fund IDs in the http response. No dice. Plan C – Selenium.

Take 2: Selenium

I used the RSelenium R package for this mini project. There are also Selenium bindings for Python, Java, C#, Javascript and Ruby which make replicating this process in your programming language of choice relatively straightforward.

Step 1: Fire up Selenium

library('RSelenium')
checkForServer() # search for and download Selenium Server java binary.  Only need to run once.
startServer() # run Selenium Server binary
remDr <- remoteDriver(browserName="firefox", port=4444) # instantiate remote driver to connect to Selenium Server
remDr$open(silent=T) # open web browser

Step 2: Start scraping

To figure which DOM elements I wanted Selenium extract, I used the Chrome Developer Tools which can be invoked by right clicking a fund in the table and selecting Inspect Element. The HTML displayed here contains exactly what we want, what we didn’t see with our http request.

Since I want to grab all the funds at once, I tell Selenium to select the whole table. Going a few levels up from the individual cell in the table I’ve selected, I see that <tbody id="tbody"> is the HTML tag that contains the entire table, so I tell Selenium to find this element. I use the nifty highlightElement function to confirm graphically in the browser that this is what I think it is.

Then it’s business as usual. I parse the string output from Selenium into an HTML tree and use XPath to parse the table for just the fund name and ticker.

library('XML')
master <- c()
n <- 5 # number of pages to scrape.  80 pages in total.  I just scraped 5 pages for this example.
for(i in 1:n) {
  site <- paste0("https://www.fidelity.com/fund-screener/evaluator.shtml#!&ntf=N&ft=BAL_all&msrV=advanced&sortBy=FUND_MST_MSTAR_CTGY_NM&pgNo=",i) # create URL for each page to scrape
  remDr$navigate(site) # navigates to webpage
  
  elem <- remDr$findElement(using="id", value="tbody") # get big table in text string
  elem$highlightElement() # just for interactive use in browser.  not necessary.
  elemtxt <- elem$getElementAttribute("outerHTML")[[1]] # gets us the HTML
  elemxml <- htmlTreeParse(elemtxt, useInternalNodes=T) # parse string into HTML tree to allow for querying with XPath
  fundList <- unlist(xpathApply(elemxml, '//input[@title]', xmlGetAttr, 'title')) # parses out just the fund name and ticker using XPath
  master <- c(master, fundList) # append fund lists from each page together
}

head(master)

## [1] "FidelityAA Global Balanced Fund (FGBLX)"                    
## [2] "FidelityAA Global Strategies Fund (FDYSX)"                  
## [3] "Fidelity FreedomA 2055 Fund (FDEEX)"                       
## [4] "Aberdeen Dynamic Allocation Fund Class A (GMMAX)"           
## [5] "Aberdeen Dynamic Allocation Fund Class C (GMMCX)"           
## [6] "AllianceBernstein Real Asset Strategy Advisor Class (AMTYX)"

Step 3: Extract ticker

Nothing fancy here – just separating the ticker from the fund name.

master2 <- data.frame(sapply(master, function(x) substr(x, nchar(x)-5, nchar(x)-1)))
master2$name <- sapply(master, function(x) substr(x, 0, nchar(x)-8))
names(master2) <- c('ticker', 'name')

head(master2)

##   ticker                                                name
## 1  FGBLX                     FidelityAA Global Balanced Fund
## 2  FDYSX                   FidelityAA Global Strategies Fund
## 3  FDEEX                        Fidelity FreedomAA 2055 Fund
## 4  GMMAX            Aberdeen Dynamic Allocation Fund Class A
## 5  GMMCX            Aberdeen Dynamic Allocation Fund Class C
## 6  AMTYX AllianceBernstein Real Asset Strategy Advisor Class

What else can Selenium do?

My little example makes use of the simple functionality provided by Selenium for web scraping – rendering HTML that is dynamically generated with Javascript or Ajax. Since Selenium is actually a web automation tool, one can be much more sophisticated by using it to automate a human navigating a webpage with mouse clicks and writing and submitting forms. This can be a huge time saver for researchers that rely on front-end interfaces on the web to extract data in chunks.

Here’s a basic example using Python.
Here’s a basic example using R.

Getting setup

On the several computers I use, I’ve found setup ranging from seamless to frustrating.

The most frustrating issue I encountered while setting up on my Mac was this error message:

> remDr$open()

[1] "Connecting to remote server"
Error:    Summary: UnknownError
 	 Detail: An unknown server-side error occurred while processing the command.
 	 class: java.lang.IllegalStateException

I was able to resolve it by killing all processes running on port 4444 and trying again.

At the terminal:

lsof -i :4444

Kill PIDs of any processes listed. For example:

kill 30681

Scraping with Selenium was originally published by andrew brooks at andrew brooks on December 11, 2014.

DIY building an R package

2014-11-20T00:00:00+00:00

Why create a personal R package?

As a consulting data scientist, I write a lot of R code in a lot of different places – physically and virtually. Different computers, servers, evironments, VPNs, operating systems, all of the above. Even when I have the luxury of working with the same client (and computing environment) for enough time to work on different projects, things can get messy.

When is it worth it?

I find myself often facing a dilemma – do I keep project specific code consolidated in one location at the expense of possible duplication later on by copying around old general purpose functions to allow for customization and further development in the future? Or do I maintain the general purpose functions which may be called from several different projects in one location at the expense of making customization and enhancing functionality more of a headache?

I find there are pros and cons to each method:

Decentralized: portable, customizable … but can be grossly duplicatitive and suffer from curse of versionality.
Centralized: organized, clean, efficient, scalable … but can be rigid, requires discipline and can break old programs if you’re not careful.

Surely centralization is the better solution after some tipping point. However, the unpredictable nature of my work sometimes makes it difficult to predict when (or if) that tipping point will occur – when the benefits of centralization begin to outweigh the costs of the portable lightweight decentralized method.

Why not just a folder full of functions?

I’m not sure I have a good answer to this yet. This was my previous solution for maintaining general purpose R functions until building a package.

Some current thoughts:

Organization: I’m finding it easier to organize my functions with structured documentation on parameters and examples, albeit with some upfront cost of actually writing this documentation.
Sharing: makes it easier to share your code that is documented in a common-tongue with others.
Version Control: If you’re using Github, you can always revert back to previous versions.
Tests & Checks: When building a package, your code is evaluated for errors and missing dependencies, including your examples.
Shiny apps: I realized during this process that you can actually wrap Shiny apps up into functions and configure them to take arguments from your environment. Useful for quick and dirty exploratory work so you don’t have to worry about directories or change ui.R and server.R code around to fit something new you want to throw in a Shiny app.

How to start building

I got started following Hilary Parker’s post: Writing an R package from scrach. This gets you a minimal package on Github.
I had to fill in with some steps from Steve Mosher’s post on building in Windows. I needed to add R to my path and install Miktex.
For anything else, you can probably find it on R guru Hadley Wickham’s R packages page soon to be published (2015) by O’Reilly .

These made my life easier

devtools R package.
roxygen2 R package
RStudio for building and testing
Miktex

Workflow for using the package

Once you’ve built a package on Github, it’s simple to pull it down wherever you are.

 
install.packages('devtools') # if not already installed
library('devtools') 
install.packages('brooksandrew/Rsenal')
library('Rsenal')

Workflow for adding to a package

Clone the git repository from Github locally on whichever machine you’re on.

 
  git clone https://github.com/brooksandrew/Rsenal.git Rsenal

Add function(s) to the \R folder of your package.
Good practice to add packageName:: before each function from an external package, so it’s clear what your dependencies are for each function.
Update DESCRIPTION file with package dependencies: imports and suggests.
Check and Build in RStudio.
Commit and push changes to Github.

 
git add .
git commit -m "adding functions to R package"
git push origin master

Gotchas

If using RStudio and roxygen2, you might have to Configure Build Tools to allow Roxygen to generate the documentation you want it to.

In RStudio:
=> Build
=> Configure Build Tools
=> Check box for “Generate Documentation with Roxygen”
=> Click “Configure”
=> Probably want to check boxes for at least “RD files” and “NAMESPACE”.

If your NAMESPACE file (simple list of your functions and dependencies) isn’t updating and you dont want to use RStudio, try devtools::document()

DIY building an R package was originally published by andrew brooks at andrew brooks on November 20, 2014.

Getting feet wet with AWS

2014-11-10T00:00:00+00:00

I’ve been intrigued by Amazon Web Services (AWS) for a while now. I spent some time this week exploring the workhorse (EC2) and the warehouse (S3) among a couple other more lightly used AWS offerings (SimpleDB and RDS).

Here’s the presentation I shared with my company during an internal tech-time with my fellow data geeks:

First impression

Admittedly, I don’t have a slew of cloud computing experience for comparison, but I was thoroughly impressed with the capabilities and usability of AWS, especially EC2. A data scientist with intermediate-ish command line skills, basic knowledge of technology, and a little bit of patience can be their own Sysadmin and DBA pretty quickly.

In the past, I always felt like I was walking on eggshells working on servers, usually just a partition of some bigger machine doing at least something important. Usually some system or network admin would handle things my user role didn’t have permissions to do in the background, which could be nice, or slow, or a total road block. Not so with EC2. They take care of all the hardware and give you the reins for everything else – the fun stuff. I found I learned a lot more with the license to be dangerous. Worst case, terminate your instance and built it again. Just don’t do it on the beefy (and pricy) 8Xlarge 32 core instance…

Getting started

I have to say I was thoroughly impressed with how easy it was to get started. The great thing about getting started with AWS is that there are minimal environment-specific issues to bog you down, since you’re really not working with your environment or your computer. The chunk of polycarbonate in front of you is just the vessel. And if you’re like me, those simple 2 line installation READMEs that go something like

get someNew.thing
install someNew.thing

work maybe 1 in 5 times without having to install some hidden environment-specific dependency, manage multiple versions of the same thing, modify your PATH, or scour StackOverflow hoping someone had the same issue requiring you to modify your specific environment somehow to accommodate the new thing.

AWS is everywhere

I was doing some research on S3 vs. Dropbox… and then I discovered that Dropbox actually uses (or at least used to use) S3 to host all your favorite kitten photos and documents you save to Dropbox.

Curious to see how slides.com, which I used to generate my slides would handle images when I exported the presentation to HTML, I inspected the exported HTML code to find that the platform automatically saves and hosts your content to S3. Each image (and other rich content, I expect) is available from a unique URL to an S3 bucket.

Making of the slides

I also took this opportunity to experiment with some new slide/presentation technologies. Having used Powerpoint and LibreOffice Impress for most of my presentations in the past, I wanted to mess around with some of the web technologies/platforms. impress.js and prezi seemed a little dramatic and flashy for my purpose. I liked the simple style of reveal.js, so I started there. The framework seemed relative straightforward and even supports markdown which I use to write this here blog, so that was cool.

However, not writing HTML and JavaScript everyday, I was spending more time tweaking code than working on content. For a more serious presentation, I definitely could have committed more time to this, but I was looking to throw together some casual slides together pretty quickly. I ended up using slides (the online graphical editor for reveal.js) which I found to be a simple yet useful and impressive solution.

Cool things about slides:

name. Seriously, no one thought of slides.com before now?
free version for public presentations.
export presentation to HTML, albeit messy HTML.
images and content hosted on S3, so HTML presentation can stand-alone (no zipping and emailing dozens of files around) if you want to share presentation via email.
simple features I used almost all the features available, but didn’t find myself desiring much more. Good balance that kept me focused on content.
intuitive UI: I enjoyed not having to navigate worlds of dropdowns, ahem Powerpoint.
code blocks: scrollable code blocks embedded directly in the presentation.
you get the code, so you can customize with reveal.js as much as you want after getting started with slides. Note: untested…this may or may not be a good idea.

Getting feet wet with AWS was originally published by andrew brooks at andrew brooks on November 10, 2014.

Deploying Shiny apps with shinyapps.io

2014-10-11T00:00:00+00:00

So I’ve been messing around with Shiny for a year or so now. It’s great tool and getting greater.

the good:

capability to rapidly build an interactive visualization with the full universe of R packages to choose from for the computation engine and visualization options.
ability to code and house the visualization/app in the same language/place as the code that actually conducts the analysis.
integrates seamlessly into analyst’s workflow.

the bad:

sharing can be hard
developed by analysts, used by analysts … and maybe others
need to download and run R to power app

the ugly:

if your app is ugly, it doesn’t have to be! Make it prettier with one of the 951,233,521 R graphics packages out there.

Option 1: Rstudio Shiny Server: If you have a linux server and the IT chops to configure it, you can set up your very own Shiny Server to host your whizbang Shiny apps. We actually set one up at my company, and it is pretty awesome… however I didn’t set it up and I don’t pay for the server.

Option 2: Shinyapps.io: If you’re just a mere human, like myself, without a server running in your house to host your app on some domain (which also costs $$), shinyapps.io is for you!

It’s pretty nifty. You can deploy all your apps to the cloud by interacting soley with R and a little configuration within the shinyapps.io interface in the web browser.
For free.

Shinyapps.io is the successor to the Glimmer and Spark hosting services Rstudio experimented around 2012-2013. I setup a Glimmer account last year (which still works as of now – October 2014). You actually interact with Rstudio in the internet browser. I never established a steady workflow… unsure if I should be developing my apps locally and then SSHing into the Glimmer server and copying the apps there, or developing on the server’s file system…

In any case, shinyapps.io makes things even easier. Simply develop and test your apps locally on your computer, and when you’re ready to deploy to the world, you simply run:

	deployApp()

Getting started with shinyapps.io

A comprehensive walkthrough can be found here. But the gist of it is pretty simple.

First register for an account here.

Then run the following R code:

require('devtools')
devtools::install_github('rstudio/shinyapps')
require('shinyapps')
shinyapps::setAccountInfo(name='yourUserNameHere', token='yourTokenHere', secret='yourSecretHere')

Then navigate to directory of app

setwd('/Users/whoeverYouAre/yourShinyApps/yourFavoriteApp')
deployApp()

And voila - you’ve deployed yourFavoriteApp Shiny App to the world… viewable at the URL something like: https://yourUsersName.shinyapps.io/yourFavoriteApp/

shinyapps.io is still very much a work-in-progress. For example, It’s currently not possible to delete deployed apps. However, for free hosting of multiple apps with an easy-to-use R and web interface and unintrusive workflow, it’s pretty good start.

###Proof that it works:

This is a visualization around the IPIP-NEO personality test. I generated some fake personalities for some familiar names (if you’re a GoT geek).

Also viewable at the URL: https://brooksandrew.shinyapps.io/ipipex/

Update (9/1/2015): It appears I’ve been bumping up against my monthly limit of free hours with shinyapps.io, so if the app below is not rendered, that’s why. If you’re hosting an app on a website like this, you can change Idle Instance Timeout time in Settings to 5 minutes instead of the default of 15 to conserve some hours.

Deploying Shiny apps with shinyapps.io was originally published by andrew brooks at andrew brooks on October 11, 2014.

Walking around DC with a camera

2014-10-05T00:00:00+00:00

Having some fun walking around the monuments at night messing around with my new(ish) DSLR and tripod.

Maine Ave SW, Washington DC. Red+Yellow+Green stoplight??!! F/16 for 10 seconds.

Jefferson Memorial, Washington DC. Obligatory DC monuments photo. F/16 for 8 seconds.

Walking around DC with a camera was originally published by andrew brooks at andrew brooks on October 05, 2014.

When Google Search fails, and when it doesn't

2014-09-28T00:00:00+00:00

What Google can’t do

Google (Search) is pretty awesome. Sometimes it knows what I want more than I know what I want. But not with punctuation, special characters, symbols, whatever you want to call them – stuff like !#$%&*+-/.,;:

When I want to search for special characters

I find myself trying to search for special characters in two common situations:

Code: I’m often googling syntax for whichever programming language I’m coding in. Code is generally littered with special characters… especially if you’re doing something with regular expressions… like ([[^:]]+)://([[^:/]]+)(:([[0-9]]+))
Writing: When I’m writing, I often use Google to help get a feel for how conventional/correct/mainstream the phrase or grammar I’m using is. If the New York Times (or any organization of prestige) uses the piece of language I’m inquiring about, then I feel good about it. If Google returns only a couple sketchy websites with visibile editing lapses, I’ll probably rephrase my language. Take “editing lapses” for example. I thought of the phrase just as I was writing the previous sentence you just read. Having not used the phrase (that I can remember) before, I Googled it. Sure enough, one of the first hits was the New York Times Manual of Style and Usage:

If the justification is lame or lacking, the correction should acknowledge reporting or editing lapses.

What special characters Google can do

Google claims to provide search support for some special characters:

This wasn’t always the case – checkout Google’s official Search blog post from March 2012. Search may support these special characters, but it does not allow (from what I can tell) to search for a specific quoted string with the character. I had decent luck with $ and %, but could barely get any queries to work with - or +.

However noticably absent are the comma, period and colon and semicolon which account for nearly all the grammar queries I’d want to run by Google.

It appears I’m not alone. Here on Google’s product forms page, a user complains of the same issue – inability to include a comma in a quoted string query.

What else Google can do

Google provides the simplicity of one search line that knows all (most of the time). There’s also the Google Advanced Search page, which is not obviously located on the main search page, where you can narrow your search by explicity guiding Google.

You can also utilize the simple search bar with basic Google Search operators. For some reason Google doesn’t document all of the operators – here is a more extensive list.

Some of my favorites are:

filetype:pdf returns just pdf documents… can replace with :pdf with :R if you’re looking for R scripts. You get the idea.
"this exact phrase" returns hits with the exact phrase in quotes
d3 -diablo returns hits on d3 without the word diablo. This was a problem for me when I was constantly searching for the JavaScript library D3.js and not Diablo 3 which admittedly consumed large part of my life for a short but intense period.
link:http://en.wikipedia.org/wiki/Catch-22 returns all the websites that link to the Catch 22 wiki page. I like to use this on new websites/blogs/online-storefronts to scan for any red flags and general clout on the web.
~inexpensive dinner returns hits for inexpensive dinners, cheap dinners, budget dinners, and any synonym of inexpensive with dinner. I’ve had mixed success with this one. A lot of the time the results appear the same with or without the tilda ~.

Where you can search for special characters

For code specific queries, the Stack Overflow community seems to favor SymbolHound. For the few queries I tested with special characters, it returned some honest results with my exact string of special characters, but only a few hits (sometimes 2, sometimes 0).

Slightly unrelated… but when you find a special character that you can’t identify in a paper, or want to copy that special character somewhere, you can search for it on amp-what.com

It’s not just Google. The other big search engines, like Bing don’t support the full gamut of special characters either. I did encounter a new search engine called Blekko with an interesting service – webgrep – that does exactly what I’m looking for. They search for specific strings of text including punctuation in the raw HTML and rank the top URLs. The only problem is that it takes 10 hours and is on reqest only. :( Cool concept though.

When Google Search fails, and when it doesn't was originally published by andrew brooks at andrew brooks on September 28, 2014.

How accurate is Next Bus IV: visualizing

2014-09-18T00:00:00+00:00

See the prior 3 posts for context:

This one takes a long time to load. It makes a d3.csv call to a 24MB file. I was surprised to find that reading in the file took all this time, but transformations that happen in the app take almost no time at all.

One week of Next Bus predictions

The scatter plot with zoom+pan+brush loaded too slow with the full dataset (24MB) of all 190,000 predictions for the week. It also made for a crowded visualization. So I took a sample preserving a consistent distance between predictions. The sampled data (2.5MB) takes every 10th prediction which turns out to be approximately a prediction every 90 seconds.

# keeping just estimates every nth prediction when time between predictions is less than 30 seconds
df$timediff[2:nrow(df)] <- df$time[2:nrow(df)] - df$time[1:(nrow(df)-1)]
df$timediff[is.na(df$timediff)] <- 0
n <- 10
df$timediffSample[df$timediff<30] <- rep(1:n, length.out=sum(df$timediff<30))
df$timediffSample[is.na(df$timediffSample)] <- 0

How accurate is Next Bus IV: visualizing was originally published by andrew brooks at andrew brooks on September 18, 2014.

How accurate is Next Bus III: getting the answers

2014-09-17T00:00:00+00:00

So now that we’ve collected data from the web and wrangled it into something useful, what can we say about how accurate Next Bus is?

This post is about the “making of the analysis” … which might be rather boring to those non data geeks (normal people). If you’re just interested in the story and the pictures, jump straight here!

I worked in R, as I usually do for most things statistical and graphical.

Our cleaned up data looks like this:

time                Minutes      VehicleID DirectionText              RouteID   TripID    hour  ID                   departure  arrival   est        err
2014-08-12 10:56:04      98      6470      South to Federal Triangle  64        6464164   10    6464164_6470         1          0         100.90513  2.905133
2014-08-12 10:56:14      98      6470      South to Federal Triangle  64        6464164   10    6464164_6470         0          0         100.73572  2.735717
2014-08-12 10:56:24      98      6470      South to Federal Triangle  64        6464164   10    6464164_6470         0          0         100.56467  2.564667
2014-08-12 10:56:35      98      6470      South to Federal Triangle  64        6464164   10    6464164_6470         0          0         100.39257  2.392567
2014-08-12 10:56:45      98      6470      South to Federal Triangle  64        6464164   10    6464164_6470         0          0         100.22040  2.220400
2014-08-12 10:56:55      98      6470      South to Federal Triangle  64        6464164   10    6464164_6470         0          0         100.04953  2.049533
2014-08-12 10:57:05      97      6470      South to Federal Triangle  64        6464164   10    6464164_6470         0          0          99.87928  2.879283
2014-08-12 10:57:16      97      6470      South to Federal Triangle  64        6464164   10    6464164_6470         0          0          99.70852  2.708517
2014-08-12 10:57:26      97      6470      South to Federal Triangle  64        6464164   10    6464164_6470         0          0          99.53807  2.538067
2014-08-12 10:57:36      97      6470      South to Federal Triangle  64        6464164   10    6464164_6470         0          0          99.36845  2.368450

est is the actual time until arrival from the time of prediction.
err is the prediction error in minutes (postive values are late buses, negative values are early buses).
Minutes is the prediction made at time.

Next Bus predictions are discrete, that is they are made in whole numbers such as 1, 2, 3 … up to 99 minutes — no decimals or seconds. Each prediction (5 minutes, 10 minutes, etc) is made many times. So we can calculate some statistics around each prediction. First I simply calculated the average prediction error.

 
mape <- aggregate(err~Minutes, df, function(x) mean(abs(x)))
mape60 <- mape[mape$Minutes<=60,]

plt4 <- ggplot(data=mape60, aes(x=Minutes, y=err)) + geom_bar(stat='identity', fill='darkblue') + 
  xlab('Prediction (minutes)') + ylab('Average prediction error (minutes)') +
  scale_x_continuous(breaks=seq(floor(min(mape60$Minutes)), ceiling(max(mape60$Minutes)), 2))
  
ggsave(filename="/png/ggmapebar.png", plot=plt4, width=5, height=5, dpi=200, scale=1.3)

Then I calculated the quantiles of actual arrival times (df$est) for each possible prediction (0, 1, 2, 3…100 minutes) using .05 increments for flexibilibity.

 

estq <- data.frame(as.matrix(aggregate(
		est~Minutes, df, function(x) quantile(x, seq(0,1,.05)))
	))

which turns out to be this (only showing predictions 0-5 minutes):

  Minutes    est.0.    est.5.  est.10.  est.15.   est.20.   est.25.  est.30.  est.35.  est.40.  est.45.  est.50.  est.55.   est.60.   est.65.   est.70.   est.75.   est.80.  est.85.  est.90.  est.95.  est.100.
       0 0.0000000 0.0000000 0.000000 0.000000 0.1702833 0.1708667 0.171920 0.340590 0.341700 0.343200 0.510850 0.512670 0.5157933 0.6817667 0.6845333 0.7331833 0.8552367 1.021637 1.031120 1.203293  2.252650
       1 0.1713333 0.6837833 0.853850 0.912685 1.0263000 1.1005667 1.197133 1.229026 1.366980 1.373926 1.536183 1.541720 1.7034033 1.7104717 1.7542533 1.8816333 2.0454300 2.058844 2.228947 2.560958  3.624817
       2 0.8558500 1.5370892 1.710678 1.880133 2.0340733 2.0560000 2.219588 2.228256 2.389870 2.398855 2.559617 2.568933 2.7259833 2.7410317 2.9015967 2.9216333 3.0806933 3.248529 3.423398 3.760646  5.174917
       3 1.3672333 2.3879967 2.568347 2.738833 2.9066067 3.0677500 3.087460 3.246477 3.271973 3.419170 3.461833 3.593877 3.6993467 3.7717700 3.9336867 4.0939833 4.1599733 4.334380 4.616890 4.962980  6.411083
       4 1.9013167 3.2562533 3.588217 3.762400 3.9303300 4.0656333 4.145327 4.281953 4.443430 4.511590 4.628600 4.785950 4.8661900 4.9807400 5.1377533 5.3032667 5.4775833 5.666597 5.984240 6.335517 15.470133
       5 2.5674667 4.1172400 4.453617 4.688427 4.9506233 5.1217750 5.205957 5.337202 5.481750 5.643520 5.803400 5.849483 5.9972100 6.1619100 6.3218600 6.4860917 6.6677600 6.863565 7.217443 7.708432 16.494933

So when the Next Bus app predicts 5 minutes:
50% of the time the bus arrives between 5.1 and 6.5 minutes.
70% of the time the bus arrives between 4.7 and 6.9 minutes.
90% of the time the bus arrives between 4.1 and 7.7 minutes.

We can visualize this using the code below to explore the confidence intervals for each possible prediction: 0,1,2,3…60 minutes. I cut it off at 60 minutes because the data gets sparse after that and the lines generally continue to flat line.

require('ggplot2')  
estq60 <- estq[estq$Minutes<=60,] # keeping just predictions 0-60
colfunc <- colorRampPalette(c("pink", "firebrick"))
cols <- colfunc(4)

plt <- ggplot(estq60, aes(x= Minutes))
plt1 <- plt + 
  geom_ribbon(aes(ymin=est.5.-Minutes,  ymax=est.15.-Minutes), fill=cols[3]) + 
  geom_ribbon(aes(ymin=est.15.-Minutes,  ymax=est.25.-Minutes), fill=cols[2]) +
  geom_ribbon(aes(ymin=est.25.-Minutes,  ymax=est.50.-Minutes), fill=cols[1]) +
  geom_ribbon(aes(ymin=est.50.-Minutes,  ymax=est.75.-Minutes), fill=cols[1]) +
  geom_ribbon(aes(ymin=est.75.-Minutes,  ymax=est.85.-Minutes), fill=cols[2]) +
  geom_ribbon(aes(ymin=est.85.-Minutes,  ymax=est.95.-Minutes), fill=cols[3]) +
  geom_line(aes(y=est.50.-Minutes), col = "black", lwd = 1) +
  geom_line(aes(y=0), col = "black", lwd=0.5, linetype='dashed') +
  xlab("Prediction (minutes)") + ylab("Prediction error (minutes)") +
  scale_y_continuous(breaks=seq(floor(min(estq60$est.10.-estq60$Minutes)), ceiling(max(estq60$est.95.-estq60$Minutes)), 1)) + 
  scale_x_continuous(breaks=seq(0, 60, 10)) +
  geom_errorbar(data=estq[58,], aes(ymin=est.5.-Minutes, ymax=est.95.-Minutes), width=2, lwd=0.25) +
  geom_errorbar(data=estq[53,], aes(ymin=est.15.-Minutes, ymax=est.85.-Minutes), width=2, lwd=0.25) +
  geom_errorbar(data=estq[47,], aes(ymin=est.25.-Minutes, ymax=est.75.-Minutes), width=2, lwd=0.25) +
  annotate("text", label="50% confidence interval", x=47-1.5, y=estq60$est.65.[47]-estq60$Minutes[47], size=2.5, hjust=1) +
  annotate("text", label="70% confidence interval", x=53-1.5, y=estq60$est.80.[53]-estq60$Minutes[53], size=2.5, hjust=1) +
  annotate("text", label="90% confidence interval", x=58-1.5, y=estq60$est.90.[58]-estq60$Minutes[58], size=2.5, hjust=1) +
  annotate("text", label="Median", x=12+1, y=estq60$est.50.[12]-estq60$Minutes[12], size=3, hjust=0, fontface="bold")

plot(plt1) #visualize plot before saving
ggsave(filename="/png/ggconf.png", plot=plt1, width=5, height=5, dpi=200)

First note from the red waves above that predictions are consistently biased conservatively. That is, buses usually (~80% of the time) arrive after their predicted time.

Second note that the something weird aboout predictions of 11,12 and 13 minutes. As one would expect, the error of predictions is less when the bus is closer (<10 minutes away). As the bus gets closer, Next Bus only has to predict a couple periods ahead rather than many. It’s well known among forecasters (and non-forecasters) that the general happenings of tomorrow is a much easier task than predicting the general happenings for a specific day decades in the future. We have more information about tomorrow.

So why are predictions made a few months (11-13 minutes) into the future worse than those made years into the future (14-60 minutes)? My first thought was that forecasts are recalibrated when they start predicting in the 11-13 minute window.

I used to forecast macroeconomic indicators for a small emerging market country when I worked at the Fed. GDP, CPI inflation, monetary policy, all that good stuff. If there’s one thing I learned from my role as a country analyst, it was that forecasting is hard and usually more art than science… at least for macro indicators. Forecasts more than a couple years out are based on, well, usually not much. However macroeconomic forecasts for the next quarter or two are more grounded. There is higher frequency and more recent (useful) data to make forecasts based on nowcasts, timeseries models, and knowledge of policy, momentum, etc.

So when presented with new information about the economy of my forecast country during the forecast period (every month or two), I often faced a choice: revise or the stay the course. Knowing when to revise is hard. One doesn’t want to overreact to information too soon just to revise a forecast in the opposite direction next time. Consumers of forecasts value both accuracy AND consistency. The balance of which is tricky – there is usually a tradeoff.

This revision behavior explains why the Next Bus errors trend downward as predictions decrease from 10 to 0 minutes. But why the spike in errors 11-13 minutes out? My second thought led me to create the visualization below which allows for an investigation of individual buses and their predictions throughout the week. Possibly a forecast bug? An idiosyncrasy of this particular bus route?

Surprisingly, the revision hypothesis appears to be dead wrong (or at least very poorly implemented). Predictions often follow a consistent trend (straight diagonal line) until some point where predictions prematurely jump to 11, 12, or 13 minutes, when in reality, there is much longer to wait.

Below: Exploring the variation around predictions 0, 1, 2, 3 … 60 minutes. Reaffirming that predictions of 11-13 minutes are the most volatile.

estvar <- data.frame(as.matrix(aggregate(err~Minutes, df, function(x) sqrt(var(x)))))

plt2 <- ggplot(data=estvar[estvar$Minutes<=60,], aes(x=Minutes,y=err, label=Minutes)) +
          geom_text(aes(Minutes,err), size=3) +
          ylab("Standard deviation of prediction error") +
          xlab("Prediction (minutes)")

ggsave(filename="/png/ggstddev.png", plot=plt2, width=5, height=5, dpi=200)

Are Next Bus predictions less reliable during certain parts of the day, like rush hour? Short answer, yes - predictions are off the mark (late) most in the morning between 8am and 10am. Note this analysis is for the 64 bus heading from a residential neighborhood (Petworth) south to Federal Triangle (downtown DC). So peak ridership and traffic and thus prediction errors along the bus route would likely be in the morning.

I’m particularly interested in assessing relative accuracy of rush hour predictions, so I’m only looking at predictions on the weekdays. The lubridate R package makes working with POSIXct dates easy. First creating dummies for weekday and extracting hour.

require('lubridate')
df$day <- weekdays(df$time)
df$weekday <- ifelse(df$day %in% c('Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday'), 1, 0)
df$hour <- hour(df$time)

Calculating the median prediction error for each prediction increment (1,2,3 … 60 minutes) by hour of day the prediction was made.

agg <- with(df[df$weekday==1,], aggregate(err, by=list(hour, Minutes), median))
names(agg) <- c('hour', 'Minutes', 'err')

which returns something like this (sample below). So the average error for predictions made between 20:00 (8pm) 21:00 (9pm) when Next Bus predicts 3 minutes is 0.59874 minutes or 36 seconds (late).

#print(agg)
hour     Minutes    err
20       3          0.59874167
21       3          0.26386667
22       3         -0.04255000
23       3         -0.25696667
 0       4          1.82138333
 5       4          0.62696667
 6       4         -0.06708333
 7       4          0.80045000
 8       4          0.96903333
 9       4          0.80225000

To make this easier to interpret and analyze, I want to reshape the results from long to wide format.

agg2 <- reshape(agg[agg$Minutes<=60,], timevar='Minutes', idvar='hour', direction='wide')
agg2 <- agg2[agg2$hour>=5,]

which creates something like this (sample):

#print(agg2)
hour     err.0     err.1      err.2      err.3       err.4     err.5     err.6     err.7      err.8
    5 0.5116583 0.5411750 0.73470833  0.7683667  0.62696667 0.3133833 0.5913083 0.8793417 1.04840000
    6 0.3423917 0.1991000 0.05491667 -0.0438500 -0.06708333 0.1321667 0.1897833 0.3459500 0.07853334
    7 0.5158500 0.8785333 0.91016666  1.0939833  0.80045000 0.9994833 0.8455833 0.9087167 0.77608333
    8 0.5109667 0.7079833 0.90148333  1.1051333  0.96903333 1.1482167 1.1714500 1.0322583 1.39443333
    9 0.5126917 0.7099667 0.91218333  0.5982667  0.80225000 0.9846500 1.2013000 1.8911333 2.78528333
   10 0.3447833 0.3663167 0.43183333  0.3377000  0.50693334 0.5863583 0.8441750 0.7526417 1.30873333

Now creating the graphic that puts it all together. I chose to not show average errors when predictions are 1,2,3 … 60 minutes separately. Firstly because the trend is the same and secondly because it’s messy. So I averaged together the median errors for predictions of 5,6,7,8,9 minutes in one goup and 10,11,12,13,14 minutes in a separate group.

require('ggplot2')

cols <- c('firebrick', 'forestgreen')
agg2$err.5.9 <- rowMeans(agg2[,paste('err.', 5:9, sep='')])
agg2$err.10.14 <- rowMeans(agg2[,paste('err.', 10:14, sep='')])

plt3 <- ggplot(agg2, aes(x=hour)) +
  geom_line(aes(y=err.5.9, color=cols[1]), lwd=1.5) +
  geom_line(aes(y=err.10.14, color=cols[2]), lwd=1.5) +
  geom_text(aes(y=err.10.14+.1, label=military2std(hour)), size=4) +
  scale_x_continuous(breaks=seq(min(agg2$hour), max(agg2$hour), 2), labels=military2std(seq(min(agg2$hour), max(agg2$hour), 2))) +
  ylab('Average prediction error (minutes)') + xlab('Time of prediction') + 
  scale_colour_manual(values=cols[1:2], labels=c('average error when prediction is 5-9 minutes','average error when prediction is 10-14 minutes')) + 
  theme(legend.title=element_blank()) +
  theme(legend.position=c(.5,.1)) +
  theme(legend.text=element_text(size=12)) + 
  theme(legend.background = element_rect(fill=alpha('white', 0.5)))
  
ggsave(filename="png/gghourmedian.png", plot=plt3, width=5, height=5, dpi=200, scale=1.3)

So predictions are wrong (late) the most between 8am and 10am on the weekdays. The errors jump around a bit throughout the day. Some of this could be the chosen frequency (hours)… it would likely be smoother if I chose 4 or 5 periods of the day spanning several hours each. Some of this could be the fact that I’m only analyzing one week of data for one bus line. Perhaps there was anomalous behavior around predictions made at 9pm for a couple buses that is driving the spike in prediction errors between 9pm and 10pm.

Analysis for next time!

How accurate is Next Bus III: getting the answers was originally published by andrew brooks at andrew brooks on September 17, 2014.

andrew brooks

Sleeping Giant Rural Postman Problem

RPP Solution Route Animation

Table of Contents

Create Graph from OSM

Adding edges that don’t exist on OSM, but should

Adding distance to OSM graph

Create graph of required trails only

Viz Sleeping Giant Trails

Connect Edges

Define OSM edges to add and remove from graph

Add attributes for supplementary edges

Viz Connected Component

Viz Trail Color

Check distance

Contract Edges

Solve CPP

Create CPP edgelist

Start node

Solve

CPP Stats

Solve RPP

Required vs optional edge counts

Solve RPP

RPP Stats

Viz RPP Solution

Create graph from RPP solution

Viz: RPP optional edges

Viz: RPP edges counts

Create geojson solution

50 states Rural Postman Problem

Motivation

The Problem

The Solution

The Approach

Table of Contents

0: Get the data

1: Load OSM to NetworkX

2: Make Graph w State Avenues only

Generate state avenue names

Create graph w state avenues only

2.1 Viz state avenues

3. Remove Redundant State Avenues

3.1 Create state avenue graph with one-way edges only

Create connected components with one-way state avenues

3.2 Split connected components

Remove kinked nodes

Viz components without kinked nodes

3.3 & 3.4 Match connected components

Viz redundant component solution

3.5 Build graph without redundant edges

4. Create Single Connected Component

4.1 Contract edges

4.2 Calculate haversine distance between components

4.3 Find minimum distance connectors

4.4 Build single component graph

4.5 Viz single connected component

5. Solve CPP

5.1 Create CPP edgelist

5.2 CPP solver

Starting point

Solve

5.3: CPP results

6. Solve RPP

6.1 Create RPP edgelist

6.2 RPP solver

6.3 RPP results

6.4 Viz RPP graph

Create RPP granular graph

Visualize RPP solution by edge type

Visualize RPP solution by edge walk count

6.5 Serialize RPP solution

CSV

Geojson

Intro to graph optimization: solving the Chinese Postman Problem

Intro to Graph Optimization with NetworkX in Python

Solving the Chinese Postman Problem

Motivating Graph Optimization

The Problem

Personal Motivation