p-value.info

The role of model interpretability in data science

2016-02-08T19:29:00.000-08:00

This is a cross post of a piece I posted on medium (Feb 1, 2016):

In data science, models can involve abstract features in high dimensional spaces, or they can be more concrete, in lower dimensions, and more readily understood by humans; that is, they are interpretable. What’s the role of interpretable models in data science, especially when working with less technical partners from the business? When, and why, should we favor model interpretability?

The key here is figuring out the audience. Who is going to use the model and to what purpose? Let’s take a simple but specific example. Last week, I was working on a typical cold-start predictive modeling problem for e-commerce: how do you predict initial sales for new products if you’ve never sold them before?

One common approach is to make the most of your existing data. For instance, you can estimate the effect of temporal features, such as launch month or day of week, using your historical data. One can also find similar products that you have sold, making the naïve assumption that the market will respond similarly to this new product, and create predictors based on their historical sales. In this case, I had access to a set of product attributes: frame measurements of Warby Parker glasses. The dataset also happened to contain a set of principal components for those measurements. (If you don’t understand what this means, don’t worry, that is the whole point of this story.) I created a model that contained both readily-understood temporal features and the more abstract principal component #3, “pc03,” which turned out to be a good predictor. However, I decided to rip out pc03 and replace it with the set of raw product attributes. I pruned the model with stepwise regression and was left with a model that had three frame measurement variables instead of just one (pc03). Thus, the final model was more complex for no additional predictive power. However, it was the right approach from a business perspective. Why?

I wasn’t creating the model for my benefit. I was creating it for the benefit of the business, in particular, a team whose job it is to estimate demand and place purchase orders for initial inventory and ongoing replenishment. This team is skilled and knows the products, market, and our customers extremely well but they are not statisticians. This is a really tough task and they need help. Further, ultimately, they — not me — are on the hook for bad estimates. If they under-predict demand and we stock out, we lose those immediate sales and the lifetime value of any potential customers that go elsewhere. If they over-predict, we invest unnecessary CAPEX in purchasing the units, we incur increased warehouse costs, and we may be left with a pile of duds. Thus, the idea of my model is to serve as an additional voice to help them make their decisions. However, they need to trust and understand it, and therein lies the rub. They don’t know what principal component means. It is a very abstract concept. I can’t point to a pair of glasses and show them what it represents because it doesn’t exist like that. However, by restricting the model to actual physical features, features they know very well, they could indeed understand and trust the model. This final model had a very similar prediction error profile — i.e., the model was basically just as good — and it yielded some surprising insights for them. The three measurements the model retained were not the set that they had assumed a priori were most important.

This detailed example was meant to highlight a few reasons why a poorer, more complex but interpretable models might be favored:

Interpretable models can be understood by less-technical (non data scientists) in the business and, importantly, they are often the decision makers. They need to understand and use the model, otherwise the work has no ultimate impact. My job as a data scientist is to maximize impact.
Interpretable models can yield simple direct insights, the sort that they can readily communicate to colleagues.
Interpretable models can help build up trust with those partners and, through repeat engagements, can lay foundations for more sophisticated approaches for the future.
In many cases, the interpretable model is similarly performant to a more complex model anyway. That is, you may not be losing much predictive power.

My team often makes these tradeoffs, i.e., choosing an interpretable model when working in close partnership with a business team where we care more about understanding a system or the customer than pure predictive power. We may use classification trees as we can sit with the business owner and interpret the model together and, importantly, discuss the actionable insights which tie naturally to the decision points, the splits, in those trees. Or, in other cases, we may opt for Naïve Bayes over support vector classifiers, again because the terms of the model are readily understood and communicated.

In all these cases, it represents a calculated decision of how we maximize our impact as a data science team and an understanding of the actual tradeoffs, if there is a loss of performance.

Of course, those tradeoffs do not always land in favor of interpretable models. There are many cases where the best model, as determined byRMSE, F-score and so on, do and should win out irrespective of whether it is interpretable or not. In these cases, performance is more important than understanding. For instance,

Show me the money! The goal is simply to maximize revenue. A hedge fund CEO is probably not going to be worried about how the algorithmic trading models work under the hood if they bring home the bacon.
There is an existing culture of trust and trail of evidence. Amazon has done recommenders well for years. They’ve proved themselves, earned the respect to use the very best models, whatever they are.
The goal is purely model performance for performance sake. Kaggleleaderboards are littered with overly-optimized and overly-complex leaners (although one could argue it is related to 1 through the prize money).
Non-financial stakes are high. I would want a machine-learned heart disease detector to do the best possible job as any type II prediction error could be devastating to the subject. Here, I believe, the confusion matrix performance outweighs a doctor’s need to understand fully how it works.

When choosing a model, think carefully about what one is trying to achieve. Is it simply to minimize a metric, to understand system, or to make inroads to a more data-driven culture within the organization? A model’s total performance is the product of the model predictive performance times the probability that the model will be used. One needs to optimize for both.

When should I hire a data scientist?

2016-01-23T12:28:00.003-08:00

This is a cross-post from a piece I wrote in medium:

Being part of the New York data scene, and especially being part of multiple VC networks, I often get asked to meet and advise early-stage startups and give my perspective on setting up the right data infrastructure and data team. Frequently, I get asked “when should I hire a data scientist?” as the word on the street is that you need a data scientist on staff. Sooner is better than later, right? They are often surprised when I say, “No, not yet. You are not ready.”

The truth is that very early startups often only have a basic data infrastructure in place to support the current business and are not ready to spend precious resources for more advanced analytics and data products such as recommenders. Focus on the foundational data first. Keep the website backend up and running, keep the signups flowing into a database table, instrument the site and track how users use your products. There’s lots to do.

You need some central transactional and analytics store that will at least scale for the next year or two. Be safe and opt for boring technology such as relational databases unless you have good reason otherwise. They are tried and tested. Boring is good. Centralize the data. Build in data quality processes. Create more robust ETLs to marshall the data. The data engineer is going to support the whole business not just analytics so is a good deal. Moreover, they are easier and cheaper to hire than data scientist.

“OK, great. We’ll hire a data engineer. And then we hire a data scientist?”

No, not yet. I would recommend hiring a data analyst first. Why? An early stage startup is probably still feeling out their business model. They are still trying to work out strategically where they should go. They are probably still seeking funding. These activities require getting answers from traditional analytics to help the founders and advisors make the right decisions and to provide the necessary information to investors. Excel will probably suffice for this work — and you can even connect it to a relational database as a data source. A good analyst will take you far. If they know SQL and they can query raw data stores directly, or can do some modeling in R, even better. Importantly for a cash strapped startup, a business analyst is probably only half the price of a good data scientist.

So at this point, you have a reasonable data infrastructure, hopefully some somewhat solid data quality processes, and you’ve met the founders’ basic data needs. Now, we hire a data scientist? Well, maybe. It very much depends on the type of business and whether a data scientist will be a central part of the business model. If you could only hire one more person for the team/rocket ship, who would provide the greatest return? That is, if a central offering of the business is a data product, a data science driven process, such as recommenders, or something similar that provide a competitive advantage then now might indeed be a good time. Maybe not. Maybe you just need another analyst. You need to have a good idea of why you need that data scientist. Don’t get me wrong. I’m very pro data science. I’m a data scientist. However, I do believe that for early stage startups at least, there can be too early for a data scientist. We are not cheap. We need data, and we are not necessarily the best people to be building out the early ETLs to get the data. Others, including software engineers, can probably do a better job, more quickly for less.

One option of course is to outsource. If you have a clear, crisp question you can essentially hand over a dataset to a consulting data scientist and let them at it. Who is going to prepare that dataset, or build an API to get the data, or provide raw access to the database? That’s right: the data engineer that you hired ahead of the data scientist.

By all means hire a data scientist but let them come into an environment where there is data ready to be mined and others to focus on vanilla business intelligence reporting and analysis and free up the data scientist to focus as much as possible on what they are good at: the fun stuff, where that unique blend of business, data, math, stats, and visualization skills can really shine.

Data is not the new middle manager

2015-06-12T20:23:00.000-07:00

In April, the Wall Street Journal published article that claimed, as its title, "data is the new middle manager" and, further, in the opening paragraph set out this bold claim:

Firms are keeping head counts low, and even eliminating management positions, by replacing them with something you wouldn’t immediately think of as a drop-in substitute for leaders and decision-makers: data.

As we say in England: codswallop! (Yes, it is in the dictionary. Think "Baloney" or "BS.") Data are replacing leaders and leadership? Really?

As you can imagine, it caused a bit of a stir within the data science field at the time. I've heard a few people mention it since, one of whom who called it a "useful meme," but I simply can't believe that basic premise. I strongly believe that

Humans make decisions.
Algorithms make decisions.
Data do not and cannot make decisions.

The article has bothered for me a couple of months, simmering away in the back of my mind. Part of the reason is that I agreed with much of the article in terms of the value of data, data tools, broad data access, and operational decision-making pushed out to the fringes. However, all tolled, the arguments presented didn't provide evidence to back up the article's major, and erroneous, claim.

It is true that

Data has indeed become more readily captured and more broadly accessible within orgs. That's a good thing.
Data tools for reporting, analysis, and data discovery are better, cheaper, and easier to use than ever before. That's a good thing.
Operational and tactical (but not strategic) decision-making can, or is, being pushed down to the front lines of orgs. Transparency via data helps achieve that. That's a good thing.

However, all these points don't lend weight that data is the new middle manager. I can’t ask data, numbers, a question: hey 6, should we switch shipping carriers? 42, you have all the answers, how much should I increase my ad spend budget? As Scott Berken puts it, "Data is non conscious: it is merely a list of stupid, dead numbers".

Data is of course a key ingredient here but its role is to augment the decision maker: humans or machines. The latter is especially interesting because I would expect its role to increase over time as we gather more data and feed it to ever better machine learning techniques on ever more powerful platforms. As I argue in my book, if you have a sufficiently stable or predictable environment and a sufficiently good algorithm that you can in fact make decisions based on data alone, without human intervention, then this is called automation, a good example of which just in time replenishment in supply chains. You should be doing that where possible. That can eliminate bodies, allows quicker, more consistent, and less emotion-based responses etc. However, this is not what is being claimed. The claim is that management positions are being eliminated because data are now acting as a middle manager, making decisions.

The author claims that the cost of data tools, once so expensive that companies could only provide them to managers, has decreased significantly such that they can now be more democratized, accessible to the front lines. That empowers those people to make informed operational and tactical decisions. Such tools and data access can also facilitate coordination among teams. One can keep abreast of what else is happening in the company and help people make decisions accordingly. However, I don't think either of these eliminate the need for true leadership, people whose job is to think strategically and to make strategic decisions, people whose job it is to inspire, align, and rally the troops. If managers are just a conduit for information and serve a coordination role, that is neither leadership nor decision making.

Better data processing tools can indeed eliminate bodies, specifically data pullers and crunchers, if instead you engender a self-service culture and everyone has the tools, skills, and access that they need. However, these bodies are not leaders or decision makers.

Organizations should be leveraging data as a strategic asset as much as possible but, ultimately, you need people to release its value.

Creating a Data-Driven Organization: Two Years On

2015-04-06T17:50:00.000-07:00

This is the third post in a series documenting the process of creating a more data-driven organization at Warby Parker. The first post covered my initial thoughts as I joined the company as Director of Data science in 2013. The second post documented progress made after one year. This next installment details progress made in my second year.

What a year it has been! I believe that we have made huge strides.

THE POWER OF THE DATA DICTIONARY
After trialing Looker, our business intelligence tool, last spring and receiving great feedback from our analysts, we made the decision to drop Tableau (at least for now, although we may get back to it for executive dashboards later this year) and instead focus on Looker.

We rolled out the tool in a fairly conservative manner over a six month period. This was not because Looker integration was difficult but we had a large number of data sources, we wanted to maintain the highest level of trust in our data, and because we needed to work with the business owners to agree upon and lock down the data dictionary. That is, set out the business logic that defined our business terms, such as precisely what constitutes a customer, how we define our sales channel logic and so on.

This data dictionary, and the consequent alignment of teams across the company, may be the most significant activity to date that has contributed to an enhanced data-driven culture. Thus, I want to go through this in some detail.

Creating and Validating The Data Dictionary
Our plan was to focus on one data source at a time, partner with the department(s) who "owned" the business logic, i.e. how the terms were defined, and who had datasets that we would validate against. They possessed Excel spreadsheets that contained raw data exported from our other systems but, importantly, in which they also had layered on derived metrics, additional logic that specified say how to handle say giftcards, exchanges, and giveaways when calculating what constituted a “sale.” The idea was that they would provide us with the business logic, we would implement that in Looker and then we would generate a dataset from Looker and compare that with the spreadsheet data, row by row, to validate. What happened was a very interesting and revealing process and is the reason that this process is so impactful to the organization.

There were a number of interesting lessons. First, unbeknownst to those teams, the business logic that they provided to us didn’t always precisely match what was actually in their spreadsheets. The reason is that these spreadsheets had grown organically over years, had multiple contributors, and contained all sort of edge cases. Thus, it was hard to keep track of the complete current logic. Therefore, the act of asking those teams to list out the logic very concretely—an actual series of IF THEN statements—turned out to be a really valuable exercise in itself.

Second, there was occasional mismatch among teams for the same metric. While the spreadsheets among different teams had originally been the same for those common metrics, they had unknowingly got out of sync. This set off very useful conversations about what the logic should be, where it should be the same, and also where and why those terms should differ. The output was a commonly agreed upon set of business logic and greater clarity and visibility about any terms that differed. For instance, our finance and product strategy teams have a different and valid perspective around the terms "bookings units," i.e. how many items we have sold. Now we are in a position to have two unambiguous, clearly-documented terms—"bookings units" and "product bookings units"—and can speak a more precise language across the company. Conversely, there were also several cases where there was a difference in definitions and the teams agreed that they should in fact be the same and they came to an agreement about what they should be.

Third, because we were using SQL during the validation process, we could easily drill down to understand the root causes of any rows that did not match. We found unusual edge cases that no-one had ever considered before, such as how some split orders are processed. When we explained these edge cases to the business owners, their reaction was often "That can’t possibly happen!" but with the evidence staring them in the face, we were able to apply those learnings to our internal processes and fix and improve our order handling and other scripts. Thus, everyone won from that activity.

Finally, some of the business logic we encountered in those Excel files was a workaround and based on the limitation of the enterprise resource planning software that generated the raw data. It was suboptimally-defined business logic. Thus, we were able to change the conversation and instead ask the business owners to specify their preferred business logic: in an ideal world, what would you like this logic to be? We were then able to implement that logic thus freeing up the teams to have simpler, cleaner, and more rational business logic that everyone could understand.

As you can imagine, this was a slow, painful process as we went through each of our many data sources working with those stakeholders to bring the data into Looker, validate it (which was the most time consuming step), and have those teams sign off on it. Those initial teams, however, saw the huge benefit to this process. They understood their own metrics better, had a centralized system that they could trust, was automated, and was locked down. Based on the benefits and great feedback that they were hearing, our CEOs made that a company priority: to get all the data into Looker, fully validated, and for analysts to use that as the primary source for all reporting and analysis. They helped us create a schedule for all the additional data sources to be included and got that necessary stakeholder buy in to do the work to define, validate, and sign off on the implemented logic.

I can’t stress enough the impact of this process on us being more data-driven. Even if we were to drop Looker today (which we don’t intend to), we would still have that data dictionary and that new alignment among all the stakeholders. It literally changed the conversation around data in the company.

To put the icing on the cake, we documented that logic in what we called the Warby Parker Data Book, an internal website with a book like interface (using gitbook.io) that lists out all our data sources, all our privacy and other data policies, and lists out that data dictionary. Everyone at Warby Parker can easily use the book to understand those terms. (This Data Book is the subject of a post on Warby Parker’s tech blog.)

Data Democracy
We now we have a suite of datasets in Looker. They can be sliced and diced with each other, the data are trusted, and are the central source of truth for the organization. Many reports are now auto-generated and directly emailed to stakeholders. For other reports, Looker is used to aggregate the data which are then exported for additional analysis or manual annotation to explain insights in the data. With Looker taking on the mechanics of crunching the numbers, this has freed up time for the analysts to spend on data discovery and analysis. Consequently, we are seeing more, deeper, and richer analyses. In addition, we are able to democratize data more than ever. For instance, Warby Parker sends out customer surveys to gather feedback about our brand, products, and experience, including within our retail stores. We now use Looker to aggregate responses that originated from each store and email them directly to the individual store leaders so that they can see and respond to what our customers are saying about their particular store. As you can imagine, those store leaders love these data and this new level of visibility.

ANALYST GUILD AND OFFICE HOURS
Switching gears and focussing on the analytics org itself, we decided that the analysts guild meetings, mentioned in the previous posts, were not as effective as they could be and we decided to shelve them for a while. They had reached a critical size in which a form of the bystander effect manifested itself. That is, the larger the group got, the less individuals wanted to help out such as volunteer to present or start or contribute to conversations—the size of the group became intimidating, especially for junior analysts. The breadth of interest and skill level of the large group meant that it was also hard to continue to find topics that were relevant and interesting to all. We decided that smaller more focussed discussions centered around a more precise topic and involving the most relevant stakeholder analysts and business owners would be a better approach. We haven’t found the right balance and process yet but is something that we are working on.

To provide additional support, I offer weekly analytics office hours, one in each of our two office buildings in New York. That is a chance for analysts to ask for help with statistics, experimental design, and in general act as a sounding board for their analysis, interpretations, and ideas. This is also helpful to me to understand what people are working on, what are their pain points, and how the data team can help.

Next on Deck
So what is coming up in terms of the analytics org? Lots of training for one. We've just had Sebastian Guttierez of https://www.dashingd3js.com/ do an in-house data visualization training session attended by a dozen of our analysts.

I am also planning to do some statistics training, not for the analysts but for the middle management at Warby Parker. You will recall from my last post that statistics training with the analysts did not work out well. Thus, my plan here is that by educating the managers and making them more demanding in the quality of analysts that they receive and the use of statistical inference—in short, making them more data literate—that will constitute more of a pull model on analysts. With me pushing from the bottom and managers pulling from the top, analysts will have no choice other than to level up.

Finally, I am working on an analyst competency matrix, a document that sets out the required skills for different levels of analysts. Thus, it specifies the level of data munging, data analysis, data visualization skills and so on that are required to jump from an analyst to a senior analyst. By providing a very clear career path, and the support to develop those skills needed to get promoted, we hope that this will make for happier, more content, and productive analysts.

More generally, I want to promote more forward thinking analyses in the next year: many more predictive models and hopefully even some stochastic simulation models for supply chain.

A BOOK
As an aside, one exciting thing that happened over this last year, at least for me, is that I decided to write a book. Based on the discussion and feedback to the previous two posts in this series, I approached O’Reilly Media with a proposal for a book (imaginatively) entitled "Creating a Data-Driven Organization" which was soon accepted. Thus, since August I’ve been more intensely researching what it means to be data-driven, interviewing others about their experiences, and writing up a long form synthesis. I’ve learned a huge amount, it has been a lot of fun, and I’m in the final stages—just revisions and corrections to do. In fact, although not quite complete, it is now available for purchase as part of their early release program.

As with these posts, I would love to continue the discussion and get your feedback and learn about your experiences. I shall be presenting on this topic at http://www.next.ml/ and at http://datadayseattle.com/.

AN INCREASING THIRST FOR DATA
Bringing the conversation back from the analytics org to the company level, I’m definitely seeing a thirst for data now. Analysts are wanting more and more data. This is a great problem to have. For instance, in my first year, analysts were doing crude, high-level geo-analyses. It had some value but they wanted more detailed insight into the business. Thus, we provided them with a dataset containing ZIP codes, CBSA (metropolitan areas), and DMA (TV viewing areas) and folded those into our customer and sales data. This set off a flurry of deeper, more nuanced reporting, which was fantastic. Last week, however, that same team approached us again and asked how they can get neighborhood level detail. With Warby Parker opening more retail stores, they wanted a finer view of the local impact of those stores.

In addition, a couple of days ago, I attended a Warby Parker management retreat, a quarterly review and planning session. One of the themes that popped up in a number of conversations was more data, more visibility, and even the term "data-driven" was mentioned many times. Good things are happening and I really sense a cultural change.

As before, check back in a year’s time to monitor our progress.

The "Creating a Data-Driven Organization" book is now available in Early Release

2015-03-21T11:48:00.002-07:00

My book Creating a Data-Driven Organization is now available for purchase as part of O'Reilly's early release program. That means you can get access to chapters as they are released before the print date in July 2015.

Another advantage is that you have the chance to input and shape the book. I would love and appreciate your feedback and comments as I have another month or so to incorporate major changes. If you have anything that you would like to add or say, feel free to add comments via the add errata link.

Many thanks

Carl

Advice to graduate students interviewing for industry positions

2015-03-09T19:58:00.000-07:00

A couple of weeks ago I saw a post in a LinkedIn group which went something like this: "I've just received a Ph.D. in physics and I know python and R. I've been applying for data scientist roles. However, I'm not getting much traction. Do you think that I need to learn a BI tool such as Tableau?" To summarize, this is a physics Ph.D., python and R! That is pretty much a trifecta for an ideal data scientist background. In the current climate, he should be inundated by offers.

I didn't interview this person but I assume that he could be doing a better job at selling himself based on his background, experience, and skill set. This is something that I have seen many times when interviewing graduate students over the years. Many students grossly undersell themselves, which is huge shame. Thus, I want to take the opportunity to give a few pieces of advice from a hiring manager's perspective.

When I've interviewed graduate students wrapping up their Ph.D.s, too many times the conversation goes like this:

Me: so tell me about yourself and what you've been doing at University of X.
Candidate: I was working in Professor Boson's lab which studies Doppler Shifts. While we have a pretty good idea of Y, we don't understand Z....[science, science, science]...and our group was specifically looking at [science, science, science]...
Me: OK but what what was your role?
Candidate: I was analyzing red shifts using geodesic equations...[science, science, science]...

(if you haven't worked it out, I know nothing about this particular area. I'm just trying to make a point.)

Don't get me wrong. I have a science background and was a professor. I love science, learning about anything new, and could chat about it all day. However, from a hiring manager's perspective, so far in this conversation, I haven't heard anything that is relevant or useful to me. My organization doesn't study red shifts. What I'm interested in are transferable skills that can be applied to different domains or the problem sets in my company. Thus, what I want to hear about are problem solving skills, coding skills, data munging skills, how you overcame a huge challenge and so on.

So, I often have to push the conversation quickly to these areas. After some probing, I might then find out that they had to process huge amounts of data from optical arrays, or they had to deal with a significant missing data problem and impute values, or they had to develop some sophisticated computer vision algorithms. They do in fact have a more interesting and marketable skill set that they, unfortunately, aren't leading with. In short, I find that graduate students often don't think about what skills they posses that are valuable to an organization to which they are applying. Draw attention to those clearly in your resume and in how you talk about yourself during a phone screen. In essence, explain why we would be a good match.

The fact that you are completing a Ph.D. shows focus, persistence, and dedication. Research is often open-ended and you have to get a sense of where the ripe questions and approaches are and know when to give up and tackle it another way. That is a highly valuable skill. Dealing with messy real world or voluminous data are problems that we face in industry. We want smart, creative thinkers who can switch domains, own a problem, think laterally to deal with unknown but inevitable issues that will crop up, and who, ultimately, still produce results. We want good communicators. We want to know what combination of those you are. Where do you shine and what do you bring to the table? And, we need to make an initial assessment of all of this in 30 to 45 minutes over a phone line. It's tough so you have to put your best foot froward. Want an insider tip? You can get more of my time if, in advance of the call, you have a github account or a competition profile on Kaggle listed on your resume that I can check out.

No, you don't need to learn Tableau to get my attention. You likely already have a great set of skills. Just sell me on what they are. What can you do?

Good luck!

Creating a Data-Driven Organization: the book

2014-10-12T14:56:00.000-07:00

I am pleased to announce that I am currently under contract with O'Reilly to write a book on "Creating a Data-Driven Organization." It is still at the early stages but should be out next year.

This is a great opportunity to go deep into what it means to be data-driven and to think about how organizations best achive that goal. I'll cover the infrastructure, skills, and culture needed to create organizations that take their data, treat it as a core asset, and use it to drive and inform critical business decisions and ultimately make an impact.

So far, it has been a lot of fun. I've read and learned a ton and can't wait to share.

No word yet on which animal will appear on the cover...

Creating a data-driven organization: the presentation

2014-09-27T18:30:00.000-07:00

Following on from my earlier posts, How do you create a data-driven organization and How to create a data-driven organization: one year on, I recently gave a presentation entitled "Creating a data-driven organization" at the Predictive Analytics & Business Insights 2014 conference in Philadelphia. You can obtain the slides here.

How data science shapes our world view

2014-06-29T19:13:00.000-07:00

Data science is increasingly impacting how we view and experience our world.

Last weekend, I was playing around with WordLens, a translation / augmented reality app recently acquired by Google. You point the smartphone's camera view at clear text such as road signs, book covers, and the like and the app not only translates the text but swaps the text out completely, matching the background (almost) seamlessly. You see that road sign in English? I see it in Spanish.

When it works well, it can be impossible to tell that the camera view, and target object, was modified in any way. Yes, this app is really buggy. It is not at all ready for prime time but nevertheless you can see the future right there in your hand. Let Google throw a team of engineers at it for the next 12 months, integrate it into glass, and you have a killer app.

One of the reasons that I am so excited and interested in this technology is that this is a significant step forward in augmented reality and, it could be argued, blurs the line significantly with virtual reality -- it is hard to tell the difference between the original world and the modified world. Moreover, it reinforces just how data science can and will influence our world view, literally.
Think of the ways that data science shape and filter what you see, sense, and perceive in the world:

Data science powers your newsfeed, showing what it thinks you want to see and who you want to interact with and filters out what it thinks is irrelevant. This obviously hugely influences your social interactions.

Skype will soon be introducing real time translation in calls. If that works well, it could dramatically shape how we collaborate with other around the world, enabling collaborations that would ordinarily be prohibitive because of a language barrier.

Adverts in the real world are becoming dynamic. In London, smart trash cans were introduced (and thereafter soon banned) that would read your mac address and target display ads at you directly.

It kind of sounds like a rogue state, an Orwellian future: a set of algorithms that effectively controls what news you see, which friends you see and interact with, translating and modify the sights and sounds around you. A large proportion then of what you see, hear, and experience could be shaped by data science. Thus, as practicing data scientists, this represents a great deal of responsibility.

I started writing this post before Facebook's newsfeed manipulation study was published, a perfect example of precisely the sort of scenario I am talking about. If you've been hiding under a rock and didn't see it, the core Facebook data science team published a paper in Proceedings of the National Academy of Sciences (one of the most prestigious science journals) about an experiment to test social contagion of emotions. To quote directly

I am old enough to remember sitting down and consuming my news sequentially, cover to cover, on pieces of paper. (Kids, those things are called "newspapers.") I had to make a decision what to read and what to skip. Importantly, I got a brief sense of the news articles that I was not reading. This is completely different today. While I read a physical newspaper for long form articles at the weekend, during the week I consume my news from a variety of sources, all shaped, curated, and filtered by algorithms: on Twitter, Prismatic, Reddit and the like. I have a higher hit rate of stories that interest me but I have no idea what I am missing, what's been buried.

"The experiment manipulated the extent to which [Facebook users] (N = 689,003) were exposed to emotional expressions in their News Feed. This tested whether exposure to emotions led people to change their own posting behaviors, in particular whether exposure to emotional content led people to post content that was consistent with the exposure—thereby testing whether exposure to verbal affective expressions leads to similar verbal expressions, a form of emotional contagion."

and they found that

"When positive expressions were reduced, people produced fewer positive posts and more negative posts; when negative expressions were reduced, the opposite pattern occurred."

As I write, there is heated debate about this study. There are those that argue that there was no informed consent of the user. Even the editor of the paper (Prof S. T. Tiske) expressed concern:

"I was concerned until I queried the authors and they said their local institutional review board had approved it—and apparently on the grounds that Facebook apparently manipulates people's News Feeds all the time... I understand why people have concerns. I think their beef is with Facebook, really, not the research." (Source: article in the Atlantic)

and therein lies the flip side. A / B tests, personalization, recommenders, coupons etc. manipulate users all the time. Is this really any different? What makes overt emotional manipulation worse that manipulating their likelihood to open their wallets and purchase a product or share your content?

Helpful hint: Whenever you watch TV, read a book, open a newspaper, or talk to another person, someone's manipulating your emotions!
— Marc Andreessen (@pmarca) June 28, 2014

I don't want to take sides here but simply want to reinforce that point that as data scientists our work influences people, real people. Keep them in mind and seriously consider ethics and informed consent. Would the user expect their data to be used in this manner? Would you be OK if it were your data? If the answer is no to either of these, then don't do it. If you look out for the customer, put them first, then the business will surely follow.

How to create a data-driven organization: one year on

2014-05-07T05:09:00.000-07:00

A switch from "I think" to "The data show"

A year ago, I wrote a well-received post here entitled How do you create a data-driven organization?". I had just joined Warby Parker and set out my various thoughts on the subject at the time, covering topics such as understanding the business and customer, skills and training, infrastructure, dashboards and metrics. One year on, I decided to write an update. So, how did we do?

We've achieved a lot in the last year, made some great strides in some areas, less so in others.

Initiative metrics

One of the greatest achievements and impacts, because it cuts across the whole organization and affects all managers, are to do with our initiative teams and how they are evaluated. Evaluation is now tied very strongly to metrics, evidence backing underlying assumptions, and return on investment.

What’s an initiative team?

Much of the work and improvements that individual teams (such as Customer Experience, Retail and Consumer Insights), requires software development work from our technology team. For instance, retail might want a custom point of sale application, or Supply Chain might want better integration and tracking with vendors and optical labs. The problem is that the number of developers, here organized into agile teams, is limited. Thus, different departments essentially have to compete for software development time. If they win, they get use of a team --- a 3-month joint "initiative" between an agile team and business owner --- and can implement their vision. With such limited, vital resources, it is imperative that the diverse initiative proposals are evaluated carefully and are comparable (that we can compare apples to apples), and that we track costs, progress and success objectively.

These proposals are expected to set out the metrics that the initiative is trying to drive (revenue, cost, customer satisfaction etc.) and upon which the initiative will be evaluated --- for example, reducing website bounce rate (the proximate metric) should lead to increased revenue (the ultimate metric). They are also expected to set out the initiative’s assumptions. If you claim that this shiny new feature will drive $1 million in increased revenue, you need to back up your claim. As these proposals will be reviewed, discussed and voted upon by all managers in the company, and they are in competition, it creates increased pressure to have a bullet proof argument with sound assumptions and evidence, and to focus on work that will really make a difference to the company. It also has an additional benefit: it keeps all the managers up to speed with what teams are thinking about and what they would like to accomplish, even if their initiative does not get "funded" this time around.

It took a few rounds of this process to get where we are now and there are still improvements to be made. For instance, in this last round we still saw #hours saved as a proposed initiative impact when the hourly rate varies considerably among employees. That is, they should be standardized into actual dollars or at least a tiered system of hourly rates so that we can compare against hours saved in a more expensive team. This visibility of work, metrics, assumptions and the process by which resources are allocated has really pushed us towards data-driven decisions about priorities and resource allocation.

ROI

While the initiative process covers the overall strategy and just touches on tactics at a very high level, what happens within a funded initiative is all about low-level tactics. Teams have different options to achieve their goals and drive their metrics, what features should they work on specifically, and when. This, too, is a very data-driven process all about return on investment (ROI). Again, there is a good process in place in which our business analysts estimate costs, returns, assumptions and impacts on metrics (this is the “return” component). While the developmental time is mostly a fixed cost (the agile teams are stable), costs can vary because they may choose to pay for a 3rd party vendor or service rather than build the same functionality (this is the “investment” component). These ROI discussions are really negotiations between the lead on the agile team and the business owner (such as head of Supply Chain): what makes most sense for us to work on this sprint. This ROI process also covers my team too, the Data Science team, which is outside the initiative process but involves similar negotiations with department heads who request work; this allows us to say no to teams for requests because the ROI is too low. By asking department heads to spell out precisely the business impact and ROI for their requests, it also gets them to think more carefully about their strategy and tactics.

Our ROI process is very new but is clearly a step in the right direction. Estimating both return on investment and justifying the assumptions is not at all that easy, but it is the right thing to do. In essence, we are switching from "I think" to "The data show"...

"Guild meetings are held to improve. They are there to ‘sharpen your saw’. Every minute you use for ‘sawing’ decreases the amount of time for sharpening" from a post by Rini van Solingen

Analyst guild

Warby Parker employs a decentralized analyst model. That is, analysts are embedded in individual teams such as Digital Marketing, Customer Experience, and Consumer Insights. Those analysts report to their respective team leads and share those team's goals. The advantage, of course, is that analysts are very close to what their team is thinking about, what they are trying to measure, what questions they are asking. The downside, however, is that metrics, processes and tools can get out of sync with analysts on other teams. This can --- and in our cases did --- result in redundancy of effort, divergent metric definitions and proliferation of tools and approaches, etc.

To compensate for these inefficiencies, we instituted a "guild," a group that cuts across the organization (rather like a matrix style organization). The guild is an email list and, more importantly, an hour long meeting every two weeks, a place for all the analysts to come together to discuss analytics, share their experiences and detail new data sources that might be useful to other teams. In recent weeks, the guild has switched to a more show-and-tell format in which they showcase their work, ask for honest feedback and stimulate discussion. This is working really well. Now, we all have a better sense of who to ask about metrics and issues, what our KPIs mean, where collaborations may lay and what new data sources and data vendors we are testing or are in discussion with. When the analysts are aligned you stand a far greater chance of aligning the organization, too.

SQL Warehouse

Supporting the analysts, my team has built a MySQL data warehouse that pulls all the data from our enterprise resource planning software (hereafter ERP; we use Netsuite) with 30-minute latency and exposes those data in a simpler, cleaner SQL interface. Combined with SQL training, this has had a significant impact in the analysts’ ability to conduct analysis and compile reports on large datasets.

Prior to that, all analysts were exporting data from the ERP in CSV file and doing analysis in Excel; that came with its problems. The ERP software has limits, so exports can time out. Excel has its limits, and analysts could sometimes run out of rows or more frequently memory. Finally, the ERP software did not allow (easy) custom joins in the data; you exported what the view showed. This meant that analysts had to export multiple sets of data in separate CSV files and then run huge VLOOKUPs in the Excel file. Those lookups might run for 6 hours or more and would frequently crash. (There is a reason that the financial analysts have the machines with the highest amount of RAM in the whole company.)

To combat this insanity, we built a data warehouse. We flattened some of those tables to make them more easy to use. We then ran a number of SQL trainings. We combined going through material in w3schools as well as interactive sessions and tutorials using simplified Warby Parker data. After analysts got their feet wet, we supplemented these with more one-on-one tutorials and help sessions, and also hosted group sessions in the analyst guild. The analyst guild was a place that individuals could show off their queries and share how quickly and easily their queries ran compared to the old ERP/Excel approach. We now have a reasonable number of analysts running queries regularly and getting answers to their questions far more easily and quickly.

In addition, this centralization of not just the raw data but also the derived measures, such as sales channel or net promoter score, means a central source of truth with a single definition. This has helped move us away from decentralized definitions in Excel formulae sitting on people’s laptops to standard definitions baked into a SQL database field. In short, people now (mostly) speak the same language when they reference these metrics and measures. While there is still work to be done, we are in a better place than a year ago.

"People want to move from a culture of reporting to a culture of analytics" - Steffin Harris

BI tooling

While writing SQL is one approach to getting answers, it does not suit all levels of skills and experience. Once you have a set of core queries, you likely want to run these frequently, automatically and share results (think canned reports and dashboards). This is where business intelligence tools come into play. While we did a good job at automating a number of core queries and reports using Pentaho Data Integration, we did not make sufficient progress (and I am solely to blame for this) in rolling out a more self-service set of business intelligence tools, a place where analysts can spend more time exploring and visualizing data without writing SQL. While we trialed Tableau and, more recently, Looker, my team did not push analysts hard enough to use these tools and try to switch from Excel charting and dashboard to report feedback. Thus, while we are currently rolling these out to production this quarter, we could have done this up to 6 months ago. Getting analysts to switch would have both created more high quality dashboards that could be easily visible and shared around the company. It would have gotten more people to see data on monitors or in their inbox. It would have also freed up more time for analysts to conduct analysis rather than reporting, an important distinction.

Statistics

Another area where I made less progress than I expected was the level of statistical expertise across the company. Having a Ph.D. from a probability and statistics department, I am clearly biased, but I think having statistical training is hugely valuable not just for the analysts performing the analysis and designing the experiments, but their managers too. Statistical training imports a degree of rigor in thinking in terms of hypotheses, experimental design, thinking about populations and samples, as well as the analysis per se. In many cases, analysts would ask for my advice about how to analyze some dataset. I would ask "precisely what are you trying to answer?", but they wouldn’t be able to express it clearly and unambiguously. When I pushed them to set out a null and alternative hypothesis, this crystallized the questions in their mind and made the associated metrics and analytical approach far more obvious.

I announced that I would run some statistical training and 40 people (which represented a large proportion of the company at the time) immediately signed up. There was a lot of interest and excitement. I vetted a number of online courses and chose Udacity's The science of decisions course. This has a great interactive interface (the student is asked to answer a number of questions inline in the video itself during each lesson) and good curriculum for an introductory course. It also has course notes, another feature I liked. I decided to send about 20 employees through the first trial.

It was a complete disaster.

The number of people who completed the course: zero. The number of people who completed half the course: zero. The problem was completely unrelated to Udacity; it was our fault. Students (=staff) weren't fully committed to spending several hours per week of their own time learning what should be a valuable, transferable skill. To truly embrace statistical thinking you have to practice, to do the exercises, and to attempt to translate concepts you are learning to personal examples such as specific datasets that you use in your job as an analyst. There was insufficient buy in to this degree of effort. There was also insufficient reinforcement and monitoring of progress from managers; that is, expecting participation and following up with their direct reports. I am also to blame for not having in-house check in sessions, a chance to go through material and cover problematic concepts.

I haven't yet solved this. What I have noticed is that a concrete need drives a response to "level up" and meet expectations. Over the last few months, our A/B tests have picked up. The business owners, the agile teams building features (including their business analysts and the project managers), as well as the particular manager who runs our A/B tests and analysis, are all simultaneously expecting and being expected to provide rigor and objectivity. That is, to run a sample size and power analysis in advance of the test, to define clear metrics and null and alternative hypotheses, to be able to explain why they are using a Chi-squared test rather than another test. Having to defend themselves from probing questions from other senior managers, ones who do have some experience in this area, and just asking the right questions is forcing people to learn, and to learn quickly. This is not ideal, and there is a long way to go, but I do feel that this represents a significant shift.

In conclusion, the business is in a far better place than a year ago. People are starting to ask the right questions and have the right expectations from peers. More decisions are based from data-backed assumptions and results than before. More people are starting to talk in more precise language that involve phrases such as "test statistic" and "p-value." More people are harnessing the power of databases to leverage its strength: crunching through large amounts of data in seconds. We are not there yet. My dream for the coming year is

more canned reports
more sharing of results and insights
more dashboards on monitors
more time spent on analysis and deep dives rather than reporting
more accountability and retrospectives such as for prediction errors, misplaced assumptions
more A/B testing
more clickstream analysis
more holistic view of the business

How will we do? I don't know. Check back next year!

Interview with Data Science Weekly

2014-04-27T18:59:00.004-07:00

I was recently interviewed for Data Science Weekly in a post entitled

Data Science & Online Retail - At Warby Parker and Beyond: Carl Anderson Interview.

It was also incorporated into a PDF collection of interviews here.

Update: It was picked up by Business Insider.

Update: And then things really jumped the shark on gothamist

Data science in e-Commerce

2014-04-19T13:10:00.004-07:00

I meet a lot of aspiring data scientists, people starting out who are often switching from academia or finance. They are all keen-eyed and bushy tailed, drawn in by the tales of advanced algorithms from Netflix, the latest competition at Kaggle or the shiny new visualization from Facebook. However, when it comes to e-Commerce, they are kind of stumped. They don't really grasp the scope of how data science can help a business that sells physical “stuff”. They get the idea of recommendation engines baked into almost every chunk of Amazon's website of course but beyond that, they find it hard to imagine how else data scientists may spend their days in such companies.

The purpose of this post, then, is a brief, almost superficial, overview of some of the different aspects of a typical e-Commerce business where data scientists can add value.

Before I start, however, I want to mention a couple of caveats:

• All of the areas below are serviced by a swathe of specialized vendors. They can do a great job --- potentially far superior than an in-house data science team because their business is so focused and specialized and their tools so developed --- but it usually comes at a price. At small company scale, an individual data scientist or team may be able to provide something that is sufficiently good to meet the company's needs or to demonstrate the need for a specialized service. At larger scale, it may make sense to build such systems in-house using the data science team.

• In the list below, there is a broad overlap between the responsibilities of a typical analyst and a data scientist. Some aspects, such as “implement a recommendation engine” are clearly in the data science camp. Other areas, such as those relating to customer insights, are usually performed by analysts. In this case, however, the data scientists may be able to help the business and analysts with more sophisticated statistical approaches (say feature reduction or unsupervised clustering of customers rather than a priori slicing and dicing based on age, gender, zip etc), in other words advanced analytics, or more programmatic approaches (e.g. use an API to pull down supplementary data).

Few individual companies will use data scientists for all of these aspects. The point here is to highlight different aspects where data scientists can and do get involved and provide some value and insight.

Recommendation and Personalization

Let's get the obvious one out of the way. Consumers are increasingly reliant on recommendations these days, whether it is for news, restaurants, bands or items to purchase. Many, if not most, e-Commerce sites have some sort of recommendation engine under the hood and it is typically the data scientist's role to help conceive the type, features, weights and in many case implement it. These engines are used for cross-sell (“you are ordering this iPad so you probably want one of these cases to protect it”), up-sell (“you have been looking at this camera, here is the next level up which is even more awesome”) and personalization. It is the data scientist's role to learn the attributes and relationships among products and when possible to learn the tastes and anticipate needs of the customers. They can then help tailor the customer's experience. This might involve changing the ordering of products in the search results or galley pages specifically for the customer.

Many of us have supermarket loyalty cards. So do some e-Commerce sites (think Amazon prime). They are a source of extremely valuable data (so much so that it may even be worth making some amount of loss on those customers). Coupons and discounts can drive new purchase behavior and provide insights for whole segments of customers not in the loyalty program itself. Those programs need to be conceived, managed and maximal use made of the data.

Product strategy

All e-Commerce sites have to tackle the questions: what should we sell, at what price and when. Data scientists can help define and optimize the product mix. In some cases, such as my current employer Warby Parker, the company may design and manufacture their products. That is, they own the whole process from produce conception to final sale to a customer. While there is a typically a product team that owns that design process, data scientists can and do help with forecasting. Is there a hole in our product mix, what should we make and when should we sell it? How many units should we order in the initial batch from the factory? When should we retire products? Analysts will typically tackle the retrospective analysis (how much did we sell, what are the duds) whereas data scientists can help with the more advanced prescriptive and predictive analytics.

Supply chain

If an e-Commerce is to sell “stuff,” it needs the right amount of the right stuff in the right place at the right time. Supply chain is a particularly complex and important part of the business. It is complex because it often involves multiple vendors and factories, significant time lags for international shipping, significant shipping costs (especially if one gets it wrong and has to expedite pallets of good to warehouses) and significant capex. Also, there can be very narrow windows of demand for a product and if you miss that window, you might be stuck with a big pile of useless inventory (think of “Happy New Year 2014” products on Jan 2). Finally, demand van be highly unpredictable and might correlate strongly with exogenous factors such as above-average weather. Ideally, e-Commerce will work with specialized vendors to handle supply chain or hire an expert in-house operations research team. However, in many e-Commerce sites, especially when small, there is plenty of scope for data scientists to perform detailed analysis and develop predictive models than can help minimize risk, inform strategy and optimize customer satisfaction.

Customer Service

A company that puts the customers first is going to have a great customer service team that handles issues, deals with returns and complaints and generally tries to keep the customers happy. These teams generate a trove of data from phone calls, instant messages and email interactions with the customers and back end systems. They also tend to be fairly metric driven: how long on average does it take to answer the phone, to resolve a case, what is the size of the case backlog etc.? Data scientists can help with predictive models and visualization. They can also use their skills with natural language processing. For instance, they could use keyword extraction and topical modeling to understand the types of complaints and issues being filed.

Fraud

Fraud is, unfortunately, very common and the strategies employed by the thieves varied and in some cases sophisticated. It can range from the use of stolen credit cards, non-returned items or items returned which are shrink-wrapped but which do not contain the original product. Again, there is potential for data scientists to develop models or monitoring or alerting systems

Hiring is tough, especially in technology where it is extremely competitive to hire good engineers (and data scientists of course). Hiring is time consuming and expensive because of the cost of recruiters, fees, and time spent interviewing. In addition, a bad hire can be counter-productive to the team or company and expensive to manage. Increasingly, companies are interested in honing their recruiting process: what makes a good fit for our company, where can we streamline the interview process, what are good discriminating interview questions and so on. Models can be used to understand attrition and retention, identify who should be rejected at the resume phase, and analyze and optimize the interview pipeline.

Customer insights

An e-Commerce site has stuff to sell but who are the people buying it? What are they interested in? Where do they live? How can we serve them better? What makes them tick? These questions are typically answered by analysts in a group akin to customer insights or, as the company scales, to specialized teams that might work within just one realm within the product space. As above, data scientists can help here with more advanced analytics (classifiers, predictive modeling, unsupervised clustering and segmentation and so on). This team is often responsible for customer surveys and so there is ample opportunity to help them with natural language processing including keyword extraction and topic modeling. (I wrote about this earlier in my post about matching misspelled brand names.)

Marketing

OK, so the site has product to sell and they know something about their customers. An obvious next question for them to ask is how do they get more customers or encourage existing customers to purchase more? Here we enter the realm of marketing. Once again, there is lots of scope for data scientists to contribute. This might range from adword buying optimization, channel mix optimization (by that I mean print vs web vs TV), ad retargeting optimization, and SEO. Most e-Commerce sites send out a lot of emails, especially if they are in flash sales. There is lot of scope for understanding, optimizing and A/B testing subject lines, content, send times and so on. (At One Kings Lane, a home decor flash sales site, we sent customers up to 17 emails per week.) There is a careful balance between reminding customers about your presence and what you offer and turning people off by being a nuisance. In many e-Commerce sites, cart abandonment rates reach dizzy heights and understanding and addressing that can pay rich rewards. Data scientists can often comprise a core part of personalization programs.

Web analytics

Another obvious area where data scientists can contribute is web analytics. How do people come to the site (this relates to SEO, search terms and referer URL analysis)? What paths do they take? When and where do they bounce? What stages of the checkout funnel do we lose most customers? How can we make the experience more frictionless, enjoyable and relevant? Which products are customers entering into our search box, which we do not currently, but should, supply? These are all areas which should be covered by a specialized analyst team as the company scales but where data scientists can help with data munging, visualization, advanced clickstream analysis, A/B testing as well as contributing data products for the site (personalization and recommender APIs).

And there you have it: a whirlwind tour of an e-Commerce site from the perspective of data scientists. It is this breadth that makes being a data scientist fun, rewarding and challenging. You get to work with a board spectrum of partners across the organization, dip into different domains, and make a difference in a variety of ways.

The data scene in NYC

2013-11-10T15:47:00.001-08:00

I love the NYC data scene. It is vibrant, healthy and welcoming.

Eight months ago I left San Francisco and moved to New York. I had come very close to taking an offer in DC and had been struggling weighing the pros and cons of DC versus NYC. I had previously lived in DC so knew the area, Virginia schools are very good and the cost of living significantly lower than the New York area, all important considerations when you have a young family. Warby Parker (my current employer) had set me up to have a chat with DJ Patil who at the time was data scientist in residence at Greylock Partners. We had an honest and open discussion of these issues and he asked, "That is a great offer but what happens if it doesn't work out? New York has a rich and vibrant data scene. It is going to be stimulating and there are lots of opportunities." He was right of course.

DC has changed significantly since I left in 2007. There are many more startups and networks for entrepreneurs and Harlan Harris and co have done an amazing job bringing together the data community under the umbrella of Data Community DC. The reality, however, is that there are relatively few opportunities. The two big tech companies, AOL and Living Social, are both a mess. There are many other data-related positions, if you have security clearance, and the majority of other positions tend to be with small consulting firms that service the government and Department of Defense.

Contrast this with New York:

New York is the place to be for advertising and media.
New York is the place to be for fashion.
New York is the place to be for finance.

The tech scene is rich too and while I know that this is a gross oversimplification there is somehow something very tangible about the startups here. They are more likely to sell real goods and services, physical goods from Etsy, Rent the Runway or Birchbox. In other words, eCommerce is thriving. In addition, under Bloomberg's initiatives, the city is investing heavily in data science and statistics per se with new institutes and a campus on Roosevelt Island.

At the O'Reilly Strata + Hadoop World conference last week there was an interesting panel "New York City: a Data Science Mecca." On the panel were Yann Le Cunn (NYU), Chris Wiggins (HackNY/Columbia) and Deborah Estrin (Cornell NY Tech). Yann Le Cunn is the Director of the newly opened Center for Data Science, a multi- and inter-disciplinary research institute that plans to churn out 50 data science Masters students per year as well as host a PhD program, all of which will have strong ties to the local tech scene. Similarly, Columbia's new Institute for Data Science and Engineering will be hiring 30 new faculty positions, taking up shop in a new 44k square foot building and have an industrial affiliate program. Finally, Cornell will be moving to Roosevelt in 2015 with a broader program than just data science that covers computer science and operations research, all skills the feature in the data science world. The panel made the point that New York is such a great place to be for data because of the density of the ecosystem. On a tiny island with great public transport you have a huge conglomeration of finance, media and advertising companies, other organizations such as Mt Sinai (who recently hired Jeff Hammerbacher---the very person who coined the term "data scientist" with DJ Patil), a suite of world class universities who are investing faculty, buildings (highly significant given than land is precious) and have both research and training foci, and finally the city itself. Yann also made the point that the density of jobs and other organizations here make it easier to attract part-time students.

The data-related Meetup scene is very strong too. (Meetup is based in NYC.) You could go to a packed and interesting data-or data-tech related meetup almost every night. DataKind are based here too. One of the most prominent data scientists, Hilary Mason (now data scientist in residence at Accel Partners) is based here. Strata just took place last week. In Fashion Week, 596 people attended "Fashion Tech: Demos & Drinks" that showcased the local fashion related tech companies. I couldn't attend. Why? Because I was attending a DataGotham event, another important data conference that brings the data community together. Later this month is pyData. You get the idea.

NYC is a fruitful mix of data-research, data practitioners and a strong data community. I am happy to be both here and at Warby Parker. Oh, and the team that I would have joined in DC recently imploded. I dodged a bullet. Thanks DJ.

Matching misspelled brand names -- the easy way

2013-09-08T20:07:00.000-07:00

On Friday, Warby Parker's Director of Consumer Insights approached me with some data. They had sent out a survey one question of which was "list up to five eyeglass brands you are familiar with." She was wanting to aggregate the data but, being free text, the answers were riddled with misspellings. Manually, it would take a lot of time and effort to resolve the variants. She asked if there was a better way to standardize them.

With a question like this, your first reaction might be to think of regular expressions. While there are common forms of misspelling such as transposed characters (ie <--> ei) and (un)doubled consonants (Cincinnati, Mississippi), this is not a tenable approach. First, you are not going to capture all of the variants, common and uncommon. Second, brand names are not necessarily dictionary words and so may not follow normal spelling rules.

If we had a set of target brands, we might be able to use edit distance to associate some variants but "d+g" is very far away from "Dolce & Gabbana". That won't work. Besides, we don't have a limited list of brands. This is an open-ended question and gives rise to open ended results. For instance, Dale Earnhardt Jr has a line of glasses (who knew?) and appeared in our results.

To get a sense of the problem, here are the variants of just Tommy Hilfiger in our dataset:

tommy helfinger
tommy hf
tommy hildfigers
tommy hilfger
tommy hilfieger
tommy hilfigar
tommy hilfiger
tommy hilfigger
tommy hilfigher
tommy hilfigur
tommy hilfigure
tommy hilfiinger
tommy hilfinger
tommy hilifiger
tommy hillfiger
tommy hillfigger
tommy hillfigur
tommy hillfigure
tommy hillfinger

Even if it kind of worked and you could resolve the easy cases, leaving the remainder to resolve manually, it still is not desirable. We want to run this kind of survey at regular intervals and I'm lazy: I want to write code for the whole problem once and rerun multiple times later. Set it and forget it.

This kind of problem is something that search engines contend with all the time. So, what would google do? They have a sophisticated set of algorithms which associate document content and especially link text with target websites. They also have a ton of data and can reach out along the long tail of these variants. For my purposes, however, it doesn't matter how they do it but whether I can piggyback off their efforts.

Here was my solution to the problem:

If I type in "Diane von Burstenburg" into google, it returns,

Showing results for Diane von Furstenberg

and the top result is for dvf.com. This is precisely the behavior I want. It will map all of those Tommy Hilfiger variants to the same website.

We now have a reasonable approach. What about implementation? Google has really locked down API access. Their old JSON API is deprecated but is still available but limited to 100 queries per day. (I used pygoogle to query it easily with

>>> from pygoogle import pygoogle
>>> g = pygoogle('ray ban')
>>> g.pages = 1
>>> g.get_urls()[0]
u'http://www.ray-ban.com/'

but was shut down by google within minutes). Even if you wget their results pages, it doesn't even contain the search results as they are all ajaxed in. I didn't want to pay for API access to their results for a small one off project so went looking elsewhere. DuckDuckGo has a nice JSON API but its results are limited. I didn't feel like parsing Bing's results page. Yahoo (+ BeautifulSoup + ulrlib) saves the day!

The following works well, albeit slowly due to my rate limiting (sleep for 15 seconds):

from bs4 import BeautifulSoup
import urllib
import time
import urlparse

f_out = open("output_terms.tsv","w") #list of terms, one per line
f = open("terms.txt","r")
for line in f.readlines():
term = line.strip()
try:
print term
f = urllib.urlopen("http://search.yahoo.com/search?p=" + "\"" + term +"\"")
soup = BeautifulSoup(f)
link = soup.find("a", {"id": "link-1"})['href']
parsed_uri = urlparse.urlparse( link )
domain = '{uri.scheme}://{uri.netloc}/'.format(uri=parsed_uri)
f_out.write( term + "\t" + link + "\t" + domain + "\n")
time.sleep(15)
except TypeError:
print "ERROR with" + term

f.close()

f_out.close()

where output_terms.tsv was the set of unique terms after I lowercased each term, remove hyphens, ampersands and " and ".

This code outputs rows such as:

under amour http://www.underarmour.com/shop/us/en/ http://www.underarmour.com/
under armer http://www.underarmour.com/shop/us/en/ http://www.underarmour.com/
under armor http://www.underarmour.com/shop/us/en/ http://www.underarmour.com/
under armour http://www.underarmour.com/shop/us/en/ http://www.underarmour.com/
underarmor http://www.underarmour.com/shop/us/en/ http://www.underarmour.com/

It is not perfect by any means. Those Tommy Hilfiger variants result in a mix of tommy.com and tommyhilfiger.com, which is Hilfiger's fault for having confusing / poor SEO. More importantly, about 10% of the the results map to wikipedia:

karl lagerfeld http://en.wikipedia.org/wiki/Karl_Lagerfeld

For these, I did

cat output_terms.tsv | grep wikipedia | awk -F '\t' '{print $1}' > wikipediaterms.txt

and reran these through my code using this query instead:

f = urllib.urlopen("http://search.yahoo.com/search?p=" + "\"" + term +"\"" + "-wikipedia")

This worked well and Lagerfeld now maps to

karl lagerfeld http://www.karl.com/ http://www.karl.com/

(There are of course still errors: Catherine Deneuve maps to http://www.imdb.com/name/nm0000366/ http://www.imdb.com/, a perfectly reasonable response. I tried queries with '"searchterm" +glasses' for greater context but the overall results were not great. With that I got lots of ebay links appearing.)

Now I have a hands-free process that seems to captures most of the variants and has trouble mostly only on low frequency, seemingly genuinely ambiguous cases. This can easily be run and rerun for future surveys. Laziness for the win! I don't even care if it fails to swap out very uncommon variants because in this case we don't need perfect data. We will aggregate the websites and filter out anything with frequency less than say X so those odd terms this process got wrong, we don't need to worry about. In other words, we care most about the first few bars of our ordered histogram.

Data scientists need to be good at lateral thinking. One skill is not to focus too much on the perfect algorithmic solution to a problem but, when possible, find a quick, cheap and dirty solution that gets you what you want. If that means a simple hack to piggyback of a huge team of search engine engineers and their enormous corpus, so much the better.

Leading Indicators: a response

2013-05-02T19:20:00.001-07:00

In an interesting thought experiment, Mike Loukides and Q Ethan McCallum asked the question

"If you’re looking at an organization’s data science group from the outside, possibly as a potential employee, what can you use to evaluate it?"

I certainly have my list of what I think are great companies and organizations for data science. Quora users have their list too but how do we know if we have don't first hand knowledge of working at all these places?

The authors provide two examples to motivate the discussion. The first I consider a kind of negative evidence: in a hotel, if they can't get a club sandwich right (the basics) they are certainly not going to get the details (such as great concierge service) right. That is, a club sandwich is a good indictor or proxy of the hotel's overall quality and level of service. The second example, I consider more as positive evidence. If a school has a great music program then it almost certainly excels in other areas too. Again, the music program is a good proxy for the school's overall program.

With this framework, they reframe the overall question:

"What are the proxies that allow you to evaluate a data science program from the “outside,” on the information that you might be able to cull from company blogs, a job interview, or even a job posting?"

They then list out 7 ideas. However, I am not convinced that many of them are evaluable as an outsider---excepting asking the questions explicitly during a job interview. Let's review them one by one.

Loukides & McCallum #1: Are the data scientists simply human search engines, or do they have real projects that allow them to explore and be curious? If they have management support for learning what can be learned from the organization’s data, and if management listens to what they discover, they’re accomplishing something significant. If they’re just playing Q&A with the company data, finding answers to specific questions without providing any insight, they’re not really a data science group.

How would I determine that as an outsider? Those projects would have to be externally visible, on a website, on a blog or published. Or, if the project refers solely to code, it would have to be open source. However, any write up is unlikely to contain information about the managerial component.

Loukides & McCallum #2: Do the data scientists live in a silo, or are they connected with the rest of the company? In Building Data Science Teams, DJ Patil wrote about the value of seating data scientists with designers, marketers, with the entire product group so that they don’t do their work in isolation, and can bring their insights to bear on all aspects of the company.

This is how data scientists should be incorporated into a organization but unless a job description explicitly says "you will be reporting to the head of {marketing,operations,customer insight,...}", you will be hard-pressed to know as an outsider.

Loukides & McCallum #3: When the data scientists do a study, is the outcome predetermined by management? Is it OK to say “we don’t have an answer” or to come up with a solution that management doesn’t like? Granted, you aren’t likely to be able to answer this question without insider information.

Self-explanatory

Loukides & McCallum #4: What do job postings look like? Does the company have a mission and know what it’s looking for, or are they asking for someone with a huge collection of skills, hoping that they will come in useful? That’s a sign of data science cargo culting.

This is certainly a valid point and a primary indicator. A buzz-word filled job ad with little focus may well indicate a company jumping on the bandwagon. Conversely, a larger company with a larger, more well-established team are more likely to have more specialized job descriptions. [examples]

Loukides & McCallum #5: Does management know what their tools are for, or have they just installed Hadoop because it’s what the management magazines tell them to do? Can managers talk intelligently to data scientists?

How would I determine that as an outsider? You might be able to question that during a job interview though.

Loukides & McCallum #6: What sort of documentation does the group produce for its projects? Like a club sandwich, it’s easy to shortchange documentation.

If the project is open source, then yes, good documentation is a great indicator.

Loukides & McCallum #7: Is the business built around the data? Or is the data science team an add-on to an existing company? A data science group can be integrated into an older company, but you have to ask a lot more questions; you have to worry a lot more about silos and management relations than you do in a company that is built around data from the start.

This is the sort of information that you may be able to glean as an interviewee.

I had a few other ideas:

Tech talks: if the company is hosting tech talks and attracting well-known and respected speakers from other high quality organizations, this is good positive evidence.
Team size: how many other data scientists work there? You can likely find this out from a LinkedIn search. If you are young, starting out, you might prefer a larger team with more potential mentors, better tools and so on. Those with more experience might prefer to blaze a trail.
Existing team members: who are they and what is their pedigree? You can check their LinkedIn profiles or personal websites but there are other ways. For instance, LinkedIn has some great data scientists. How do I know they are good? They tweet relevant information, they post thoughtful posts. The speak at conferences. Their team is highly active in the general data science scene. All this visibility---provided the content is meaningful---is all good evidence.
Publications: academic journal publications may or may not be a good indicator. There is typically a big gulf between academic systems and toy problems and the mess and noise of real world systems and data. An algorithm may work great on the static data set that the grad student has worked with for 3 years but it might not scale, might require far too much parameter tuning in the real world. There are many exceptions of course. It really depends on the system.
Patents: patents coming out of an organization may or may not be a good indicator. It is essentially stales data as patents reveal the degree of innovation at the company two or more years ago (given the time it takes to process and award). A strong patent culture might mean that the IP is locked down so that you may not be able to discuss systems in development at conferences, publish work, open source the code etc.
Internship program: if the company has a strong internship program, attracting top young talent from top schools, and those interns go on to do good things as data scientists, this is very good evidence.

How do you create a data-driven organization?

2013-04-10T13:11:00.006-07:00

Something that I've been thinking a lot about recently is how do you create a data-driven organization? A lot of companies play lip service to the idea but when it comes down to making decisions they end up being made by those that are more senior (by HiPPO: highest paid person's opinion) or, worse, loudest, based on gut instinct, experience or opinion. What does it take to create a company that makes evidence-based decisions and involves a broad swathe of employees vested in data capture, metric design and analysis?

I have recently taken on this challenge. A few weeks ago, I switched coasts and companies to head up the data team at Warby Parker in New York. This is a company that has been very successful to date and has grown very rapidly. So fast in fact that it has had little time, bandwidth or experience to put in place a centralized, standardized and scalable data infrastructure with professional business intelligence tools. Analysts are working at the limits of Excel and have great difficulties tying together different data sources across the company. What it does have, however, is a strong desire to change and is willing to provide resources and support to establish new data capture, analysis and reporting systems---and promoting the appropriate culture---that will take the company to the next level.

In this post, I wanted to set out what I've been thinking and how I've started to go about it. It is absolutely work in progress. I cannot guarantee that the following is the right approach or that it will go exactly as planned. Thus, I would love to hear how others have fared and what comments and suggestions that you might have.

UNDERSTAND THE BUSINESS AND INTERNAL CUSTOMERS

Listen to people. Chat with the different heads of departments, analysts and other data stakeholders and listen to what they do, how they do it, the data that they work with and what they would like to do. Ask them how the data team can help. Identify tasks that are highly manual, repetitive and could easily be automated. Identify different data sources, pain points and aspirations. Asking about what they would like to do but cannot is as important as asking what they currently do.

Identify easy wins and build trust. While the rule is always under promise and over deliver, it is always good to identify easy wins that provide some immediate benefit and set up good will and trust. We were able to identify and implement a simple javascript/HTML tool that will save one of our teams at least 100 hours/year. While it was not strictly a data project, the cost to us was just 3 hours and that team now loves our data team and will likely be more accepting of interruptions to their work flow as we implement changes.

TRAINING AND SKILL SETS

Identify workers with skills but not tools. One of our staff knows SQL well but has not had access to databases in his current position. Thus, he is relegated to working with Excel spreadsheets. Try to get those people the tools they already know how to use well and will use. There has to be some level of control here---you don't want too much tool / language proliferation and fragmentation---but if these will become core team skills or tools anyway, get them to those people early.

Identify staff hungry to learn. Identify any particular staff that are hungry for new tools and to learn new skills. These may be staff who are already taking statistics or data science classes outside work. Mentor them, work to get them tools they will make good use of. Send them to training. These people, as well as being more productive and happier, will become your prime advocates. They will be willing to mentor others and share their experience and skills.

Train and mentor. At a broader level, if all your analysts are using Excel, train them to also use SQL, R, python or some other skill to take them to the next level, some skill that will allow them to produce more detailed, insightful, automated analyses. Start with a small, motivated group and let them set a great example for others. Statistics is not only a set of tools for analysis but it also provides a framework for critical thinking, in this case about evidence. At Warby Parker, we are planning to send a reasonably large cohort of staff to statistics training this year. With great free online courses available now, this represents a relatively low cost to the company, other than employee time, but we expect the effect of having a significant fraction of the company thinking more critically, numerically and objectively, it will have a profound effect on culture and decision-making.

Carefully choose the right tools. Clearly, if you are introducing a new tool to a team or organization, make sure that it is the right one. It should perform the tasks you need, ideally with a easy to use interface but also with power-user functionality, be well documented and supported and in an ideal world be open source.

DATA INFRASTRUCTURE

It goes without saying that you a need a robust, scalable data infrastructure.

Centralize data where possible. This is very company-scale dependent but try to create a data infrastructure to bring together all the different data sources where possible to allow a holistic view of your customers and business. For instance, make it easy to tie ad strategy to clickstream to sales to social etc. A particular solution may not scale indefinitely as the business and data grow. For instance, you may need to switch from a single MySQL instance to a hadoop-based solution in time but scaling issues are always good problems to have.

Create an open data warehouse. Create a data warehouse with broad access and easy to use tables. For instance, there may be some key concepts that are frequently used for analysis that require highly complex joins across multiple tables. For those, denormalize the tables to allow easy querying (as well as other housing benefits). There will be some data that are sensitive---credit card transactions, any medical or HIPAA compliant data etc---but favor openness wherever possible.

Automate where possible. If I think that I will need to do a task two or more times, I will attempt to automate it if possible. Whenever you think something is a one off, it almost certainly is not. By automating processes, you will free up future analyst time to focus on, you know, analysis.

Focus on the team's ROI. Like everyone else, a data team has limited time and resources. Focus on return on investment. Implementing two "good enough" solutions for problems in a week may pay higher dividends than one "almost perfect" solution for one problem. Beware of diminishing returns.

Suck down data now. Some data are potentially valuable or useful but ephemeral. For instance, Instagram photos are only available for a week or so before they disappear forever. Suck them down now as you will never know when or for what you might need them for later.

METRICS AND DASHBOARDS

The goals of the the above strategies is to capture the data and make it accessible. Now comes the fun part: analysis and reporting.

Design metrics carefully. They should be unbiased, deterministic and should reflect true measurable variables. They should be readily interpretable. They should reflect the business. Design or identify metrics that make the company tick. Think carefully about units. If you end up trying to compare apples and oranges, is there some common currency, such as dollars, that they can be converted to? For instance, if you improve operations and can ship product to customer one day faster, what is that worth? Can you assign a dollar cost to per customer / per day / per order ship time?

Remove redundancy. Dashboards should be information rich. Like building a statistical model, if you have two very highly correlated metrics you can consider one to be redundant and you may do better to remove it and increase the information density of the remaining metrics.

Tailor to your audience. In some cases, it may make sense to have multiple reports with different levels of details for different audiences. For instance, a manager may have a highly detailed report about their team and responsibilities, a higher level report goes out to her team and the C-level execs get the 50,000ft view, which if you choose the right metrics, will still be useful and relevant.

Pander to the C-level. In terms of driving a data-driven culture, impressing C-level execs with dashboard and reports that deliver huge value (and are not just eye candy) will almost certainly produce a trickle down effect. They will expect such reports and will provide resources to create reports commonplace. Create dashboards so relevant that C-level execs watch them like a hawk.

Identify metrics that map to the organization's core values. One of Warby Parker's core values is to deliver insanely great customer service. Thus, there are metrics that relate to that, net promoter score being one. Those metrics should be highly visible across the organization: in top level dashboards, on screens, on reports that get emailed out.

Conversely, take out distracting metrics. One of Marisa Mayer's first actions when she took over Yahoo! was to take the share price off their internal home page. It is her job to worry about that but the rest of the org had been focusing on actions that tried to drive up the share price (unsuccessfully) and they had almost forgotten about the users and the value that Yahoo! delivered or should be delivering to them.

Where possible, tie those key indicators to other metrics that fundamentally drive it. For instance, suppose that the major component of dissatisfaction is late shipping then promote that metric in a top level dashboard.

Let the data speak. In some cases, a more machine learning or unsupervised learning approach may bring some surprising insights. For instance, many companies segment their customers using some set of subjective a priori criteria. Running unsupervised clustering may support those segment choices but it also may provide some interesting insight into new types of groups you might never have expected. Be open to findings that challenge your intuition or understanding of your business, market and customers. Be objective: if an A/B test shows a higher average order value but the results are not statistically significant accept that they are not significant. Never fish for significant results.

Let the interns speak. A data-driven organization should give the data a voice from wherever that derives. Thus, a new intern who has been analyzing data with a fresh perspective should be given as much voice and respect as a senior manager. Data are the senior partners here. Give people a voice, forum and opportunity to provide data-backed evidence.

Share the data and findings widely. A data-driven organization should share data and reports widely. This is not to say that they should be blasted to everyone as spam but those that are interested should have access. (Remember that interesting insights and alternative points of view could come from anywhere in the company.) Owners and higher managers should be open to questions, to alternative evidence, and to implement change based on that evidence.

Those are my initial thoughts. I will report back later in the year, perhaps at DataGotham in New York, about what worked and what did not. Again, if you have any suggestions or feedback, I would love to hear from you.

When tf*idf and cosine similarity fail

2013-02-19T09:38:00.000-08:00

In this post I cover 2 edge cases of cosine similarity with tf*idf weights that fail, i.e. that don't provide the cosine similarity values that intuition and common sense says that they should return.

In information retrieval, tf*idf forms the basis of scoring documents for relevance when querying a corpus, as in a search engine. It is the product of two terms: term frequency and inverse document frequency.

Term frequency is the frequency of some term in the document, typically an absolute count or relative frequency. Documents with more mentions of a term are more likely to be relevant with respect to that term. For example, when querying for "dog," a document about caring for your dog which mentions "dog" 46 times is more likely to be relevant than a document with a single mention of "the dog days of summer."

Inverse document frequency (IDF) measures the dispersion of that term across the corpus. If every document contains "the," then "the" is not a particularly discriminating word. IDF is the ratio of the corpus size to the number of documents containing that term. The smaller the proportion of documents containing that term, the higher the magnitude of this metric. (In reality, we take the log of the ratio. That is, idf = log(N/n_i)).

These two measures quantify the frequency within a document and the relative rarity across the corpus. Taking the product we arrive at a simple, satisfyingly intuitive but surprisingly powerful metric to score documents. For each term t in each document d in some corpus D we can compute the tf*idf score. Let's call this tfidf (t,d).

We rarely query a corpus for a single term. Instead, we have a query q consisting of multiple terms. Now we want to compute the similarity between the query q and each document d in the corpus. For this, we tend to use something called cosine similarity.

This is a measure of the angle between two unit vectors:

similarity

= cos(a,b)

= dotproduct(a,b) / ( norm(a) * norm(b) )

= a.b / ||a|| * ||b||

[Definition: if a = (a1,a2,...,an) and b = (b1,b2,...,bn) then a.b = Sum(a1*b1 + a2*b2 + ... + an*bn)

and ||a|| = sqrt(a1^2 + a2^2 + ... + an^2) and ||b|| = sqrt(b1^2 + b2^2 + ... + bn^2). ]

The smaller the angle, the more similar are the two vectors.

In this case, the variables of a and b are the set of unique terms in q and d. For example, when q = "big red balloon" and d ="small green balloon" then the variables are (big,red,balloon,small,green) and a = (1,1,1,0,0) and b = (0,0,1,1,1).

Not all words are created equally. Some are more important than others when computing similarity. Rather than use the count or the presence/absence of each term, we can use a weight. For example, we can give a lower weight to common words. What would make a suitable weighting? tf*idf of course. Putting this altogether,

similarity(q,d) = a.b / ||a|| * ||b||

where

a = (

tfidf("big",q),

tfidf("red",q),

tfidf("balloon",q),

tfidf("small",q),

tfidf("green",q)

)

and

b = (

tfidf("big",d),

tfidf("red",d),

tfidf("balloon",d),

tfidf("small",d),

tfidf("green",d)

While cosine similarity with tf*idf works well, really well, there are a couple of edge cases where it fails, corner cases that don't seem to be covered in most introductory explanations and tutorials.

FAIL 1: imagine you have a corpus D consisting of one document d. You come along with a query q where q == d. That is, the corpus has exactly what you are looking for. Intuition should say that we expect that cosine similarity would be 1 because q == d. So, what do we get? While the dot product of q and d should be 1 giving cosine similarity 1, it is not when you use tf*idf weights. The tf*idf of each term of d will be zero--each term of d is in all documents (D==d). Therefore, the dot product is zero but the norms of the two vectors is also zero and will generate a division by zero error. In summary, similarity is 0/0 and so undefined.

FAIL 2: imagine you have a corpus with two documents, d_1 = "blue bag" and d_2 = "green bag". What is their similarity? Intuition says there are some similarities between them, they both contain "bag," but there are some differences: "blue" vs "green". Thus, this should mean that we get a cosine similarity somewhere between 0 and 1. Wrong! Tf*idf for "bag," the common term, is zero because IDF is zero. "blue" is not a shared term and so that term of the dot product is zero as is for term "green." In other words, where they differ it pumps zero terms into the dot product and where they are similar, those terms effectively convey no information whatsoever and so also generate zero values.

While these two scenarios may seem contrived, I encountered them while writing unit tests where I wanted to use minimal corpora possible to test my code. It seems that one needs three distinct documents to avoid the problems above, or your code must handle a NaN.

I use tf*idf and cosine similarity frequently. It can get you far with little cost (if your documents are not enormous). It does have a big limitation though, it is a "bag of words" model meaning it does not consider word order. In many cases, specific word order matters a lot---a red couch with gold legs is very different from a gold couch with red legs. What one can do is to use the fast and cheap cosine similarity with tf*idf weights to narrow down some larger corpus to a smaller subset of documents for which you run a more computationally expensive, more domain specific model or algorithm that does consider word order.

20 Tips and tricks working with data

2013-02-14T16:22:00.000-08:00

In this post I lay out a number of tips, tricks and suggestions for working with data, from collection to storage to analysis. The main idea is to help students and other younger analysts and data scientists. However, they apply to all ages and experiences. Indeed, while many of the following ideas may sound obvious and trivial, it is surprising how often I see the counter-examples and anti-patterns.

This is almost certainly a biased selection based on my personal experience over the years. Thus, if you have any additional comments or tips, feel free to add those in the comments.

This section is about data itself.

1) Inspect your data.

When you receive a dataset, open a text editor and inspect the data. Don't assume all is present and correct. Page through the data quickly to get a sense of columns that are mostly NULL, data that are zero or negative, the ranges of dates and so on. It is very easy to make mistakes when reformatting data and data entry people are often low-paid, low-skilled workers. Assume that there is something wrong with your data. In *nix, use the head and more commands to get a sense of the data.

2) Think through what you would expect.

Related to above, it helps to have an expectation of what you might expect in each column. This will help in spotting errors. For instance, would you expect to find negative values? I once worked with a government-recorded health dataset that included data of people's height, something one might expect would be easy to measure and record. However, we found a chunk of people listed as 5 inches tall. Likely, they should have been 5 feet tall but such outliers are easy to spot with a frequency histogram. In R, get in the habit of calling summary() on your dataset which provides a 5-number summary of each variable. Plot univariate distribution and in R pairs(), a function that produces a scatter plot of each pair of variables, can be very valuable.

3) Check your units.

It may sound obvious but check your units. CNN reports

NASA lost a $125 million Mars orbiter because a Lockheed Martin engineering team used English units of measurement while the agency's team used the more conventional metric system for a key spacecraft operation.

Yeah, it's that important.

4) Plain text > binary.

Plain text data files are always better than binaries. You get to see the raw data, to see how columns are delimited and to see which line ending types are used. Any text editor can be used to view the data. With a simple text file, it can be easy to write a python script to iterate through the data.

5) Structured > unstructured.

While there has been a recent focus on NoSQL technologies that store schemaless documents, if data are or should be well-structured or have a strict, static schema, keep them that way. It will make querying, slicing and dicing much easier later.

6) Clean > more > less.

Much of the time building models, classifiers and so on is spent cleaning data. It takes a lot of time and can be depressing to see huge swathes of data thrown away or excluded. It is better to start with clean data. This means having quality control on data entry. Run validators on HTML forms where possible. If you can't get clean data, at least get a lot of it (assuming it is a well-structured experiment). Clean data are better than more data which is better than less data.

7) Track data provenance.

Provenance means where data came from. In many cases, provenance is is explicit as it comes from a single source or there is some source/vendor code or name. Imagine though you create a corpus for a model in which you combine data from multiple sources. If you later find a problem with one of those souces, you will likey want to back out those data or weight them less. If your corpus is just a big set of numbers and you don't know which data came from where, you now have polluted low-quality data.

8) Be honest about data loss acceptability.

When processing data, there can be problems especially when working with remote services. Servers can be down, databases connection can momentarily drop etc. Think through how important it is to process each item. If you are a bank or many other transaction based businesses, everything must work perfectly. This though comes at a high cost. However, if you are running say sentiment analysis on a twitter stream, or running some summary metrics on historical data, a small data loss may be perfectly acceptable. Will the extra cost of getting every last tweet make a big difference to your analysis?

9) Embrace the *nix command line.

It is hard not to stress this enough. Learn to love the command line. While the commands may seem terse and obtuse, you will be amazed how much more productive you can be with just a few basic commands such as:

wc -l filename: returns the number of lines in the file.
head -n filename: shows the first n lines of a file
grep searchterm filename: filter in the lines containing this search term.

10) Denormalize data for warehousing.

There are very good reasons for normalized databases tables. A well-designed schema will be flexible for future data and grow by rows not columns over time. For wareshousing where data are stored for longer term, it can often be a good idea to denormalize data. This can make the provenance more explicit, can make it easier to split into chunks, and it is often easier for analysts to work with as you can avoid costly, complex joins.

This section refers mostly to processing data with scripts

11) Learn a scripting language.

Python, ruby, it doesn't really matter which but pick one. You want a language with at least simple I/O, a range of data structures and libraries to make HTTP requests. You will end up doing a lot of file manipulation in your career and so you need a good general tool to automate tasks. Also, some analyses, especially involving time series of individuals, are far easier to do in scripting code than with SQL self-join queries.

12) Comment your code.

This is a general tip that applies to all source code not just scripts for analyzing data. When your head is in a problem there are subtle nuances about why you're doing something the way that you are. You'll forget that in a week, especially if you work on multiple projects simultaneously. Comment on why you are taking a certain approach or why this is an edge case. Focus less on what you are doing--ideally your code will be readable enough to show that---but a few such comments, in combination with well chosen function and variable names, will help you, or someone else, get into the code later. Comment for your future self.

13) Use source control.

Again, a general coding tip. Source control (CSV, SVN, git, mercurial etc.) is indispensible. First, it is a form of backup. Second, it allows one to share code and work collaboratively. Third, it allows one the freedom to experiment more and try things out without fear of getting back to your last iteration--you can always revert back to the old code if it doesn't work out. Don't leave home without it. Also, always always write a commit message when committing. Like code comments, commit comments should focus more on what new code does rather than how the new code works. Again, comment for your future self.

14) Automate.

This is obvious but is a lesson that most people learn only after they have had to do a task manually several times when they thought it was a one off. If there is any chance that you might need to do a task twice, it probably means you will actually need to do it five times. Automate the task where possible. This might mean a python script, a SQL stored procedure, or a cron job but it most cases, it is worth it. Even if only some of the steps are automated, it can make a huge difference.

15) Create services.

In years of creating data-related tooling, I've found that people always use the tool in unexpected ways. Make the tools flexible as much of possible. Doing so, will provide an ecosystem of functionality. Create top-level functions that serve the 80% of uses as well as provide access to the lower-level functionality. Where possible make the functionality as web services that can talk to each other in a simple clean manner. In many cases, JSON is preferable to XML or SOAP.

16) Don't obsess over big data.

I might get lynched for this one but there is a lot of hype over big data as service companies vie to take a slice of the analytics pie. While big data certainly has a place, many people who think they need big data don't have big data and don't need big data technologies. If you work in mega- or giga-bytes, it is not big. Take a deep breath and answer honestly, do I have big data or do I need to scale to huge proportions; do I have a problem that can be parallelized well (some problems don't); do I need to process all my data or can I sample? I should stress that I am in no way against big data approaches---I use them myself---but it is not a panacea. Other, simpler approaches may get you what you need.

17) Be honest about required latency.

In most websites, data are replicated, warehoused and analytics performed offline. That latency for analysis can often be hours or a day without any real impact on the business. There are cases where more real time analysis is required, for instance collaborative filtering or computing real time twitter trends. Such short latencies comes at a cost, and significantly more so, the smaller the value. Don't waste money creating a system capable of 1 second sales data latency if you only analyze it in batch mode overnight.

I am hesitant to dip into data visualization as that is a whole post in itself but I will include the following two:

18) Use color wisely.

While neon colors might be in vogue at American Apparel this season that doesn't mean I want to see it in your charts. Take note of default color palettes in programs such as Excel and libraries such as ggplot2. A lot of thought, research and experience have gone into those. If you break the rules do it for good reason and for good effect. P.S. don't use yellow text on a white background. While it might look good on your laptop screen in almost all cases, it is unreadable when projected.

19) Declutter charts and visualizations.

When producing charts, infographics and other data visualizations, think very carefully about what you want the reader to take away from the data. Focus on that and make that message shine through with the least amount of reader effort. Think about information density. If I need a magnifying glass to decipher all the details, you're doing it wrong. If a chart has a lot of textual annotations, you are likely doing it wrong. If you need 6 charts to convey one main point, you are probably missing some key summary metric. Use larger fonts than you might think if you are projecting to a large room. (Where possible, actually stand at the back of the room and check readability of your presentation.) You would do well to read Edward Tufte's earlier books.

20) Use # dashboard users as a metric in your dashboards.

In his foundation video series, Kevin Rose interviewed Jack Dorsey (CEO Square and Exec Chairman, Twitter) [video]. Dorsey made a very interesting point in how he has tried to make Square data-centric. A dashboard is useless if no one is using it. How valuable a dashboard or other metrics are should be reflected in the very dashboard itself: use the numbers of users as a metric in the dashboard itself. I would add that such metrics should be honest signals. That is, if reports are emailed out to recipients, the number of recipients is not necessarily a true reflection of use. Perhaps most people don't open it. Email open rate, however, is a more reliable and reflective indicator.

There you go. A few tips and suggestions for working with data. Let me know what you think and what I missed.

When is a meat sandwich like a merchant? A python joke generator

2013-01-22T09:21:00.001-08:00

When is a meat sandwich like a merchant? When it is a burgher.

Yes, you can groan but don't blame me, heckle the computer.

I enjoyed a recent New York Times piece, A Motherboard Walks Into a Bar ..." on how and whether computer can learn what is or is not funny. I'm a big fan of groan-inducing puns and Physics particle X walks into a bar type jokes. As I read the article, it occurred to me that there must be some simple lexical patterns that a computer could pick up on and auto-generate jokes. Consider the following:

What do you call a strange market? A bizarre bazaar.

That has the structure "What do you call a [Adjective1] [Noun1]? A [Adjective2] [Noun2]" where [Adjective2] and [Noun2] are homonyms and [Adjective1] and [Adjective2] and [Noun1] and [Noun2] are synonym pairs.

(A homonym is a word pronounced the same as another but differing in meaning, whether spelled the same way or not. Example: hare and hair. Synonyms as two or more different words with the same meaning. Example: lazy and idle.)

If we take a look through a list of english homonyms, we can easily pick out such joke material:

suite: ensemble

sweet: sugary

leads to "What do you call a sugary ensemble? A sweet suite."

Similarly,

What do you call a breezy eagle's nest? An airy aerie.

What do you call a coarse pleated collar? A rough ruff.

Another structure is when the homonyms are both nouns:

stake: wooden pole

steak: slice of meat

leads to "When is a slice of meat like a wooden pole? When it is a stake."

(Slightly more complicated is "When is a car like a frog? When it is being toad?")

This suggests that we can easily auto-generate jokes such as these. So, let's do it.

First, I downloaded that homonym webpage and parsed the HTML using the python BeautifulSoup library to extract the homonyms. There is one short function to parse the HTML to obtain two homonyms and their short definitions, and for each homonym I call a second function function which calls a unofficial google dictionary API to obtain the part of speech (noun, adjective etc.) of the homonym. Calling python extract_homonyms.py > processed_homonyms.txt processes a flat text file of the six pieces of information: homonym1, definition1, pos1, homonym2, definition2, pos2

Here is the code.

With the hard work out the way, generating the jokes is simple. A second short script, generate_jokes.py, has two type of jokes: 1) one homonym is an adjective and the other is a noun, 2) both homonyms are nouns:

def indefinite_article(w):

if w.lower().startswith("a ") or w.lower().startswith("an "): return ""

return "an " if w.lower()[0] in list('aeiou') else "a "

def camel(s):

return s[0].upper() + s[1:]

def joke_type1(d1,d2,w1,w2):

return "What do you call " + indefinite_article(d1) + d1 + " " + d2 + "? " + \

camel(indefinite_article(w1)) + w1 + " " + w2 + "."

def joke_type2(d1,d2,w1,w2):

return "When is " + indefinite_article(d1) + d1 + " like " + indefinite_article(d2) + d2 + "? " + \

"When it is " + indefinite_article(w2) + w2 + "."

data = open("processed_homonyms.txt","r").readlines()

for line in data:

[w1,d1,pos1,w2,d2,pos2]=line.strip().split("\t")

if pos1=='adjective' and pos2=='noun':

print joke_type1(d1,d2,w1,w2)

elif pos1=='noun' and pos2=='adjective':

print joke_type1(d2,d1,w2,w1)

elif pos1=='noun' and pos2=='noun':

print joke_type2(d1,d2,w1,w2)

print joke_type2(d2,d1,w2,w1)

When we run this, we output 493 wonderful, classy jokes (from just 70 lines of code). A few of my favorites are:

What do you call an accomplished young woman? A made maid.
When is a disparaging sounds from fans like a whiskey? When it is a booze.
When is a fish eggs like a seventeenth letter of Greek alphabet? When it is a rho.
When is a bench-mounted clamp like a bad habit? When it is a vice.
When is a fermented grape juice like an annoying cry? When it is a whine.
When is a location like a flounder? When it is a plaice.
What do you call a fake enemy? A faux foe.
What do you call a beloved Bambi? A dear deer.

Not bad, not bad although even Carrot Top's career is probably safe with these.

This is the complete source code.

(Another potential joke pattern comes from "What is the difference between a pretty glove and a silent cat? One is a cute mitten, the other is a mute kitten." where we can observe a transposition of the first letters of two pairs of words. You can discern some other patterns in this joke generator site.)

So, we can conceive that a computer could be programmed with, or learn, the structure of jokes. This is a generative approach (e.g., Manurung et al.).

A second approach is to learn which jokes are considered funny by humans. Given a suitable corpus and a reasonable set of features, any number of classifiers could learn, at least statistically, to sort the funny from the unfunny (e.g., Kiddon & Brun, That's what she said detector).

Finally, given a set of jokes, a system could learn which are funny to you given some basic training. Jester is a system where you are asked to rate 10 jokes. After that, you are presented with a series of jokes that you are more likely to find funny than other jokes. In web terms, it is an old site with what amounts to an early recommender system (Goldberg et al. 2000).

One final joke from my code:

What do you call a least best sausage? A worst wurst.

Ba dum dum, Thanks, folks! I'll be here all week.

What's the significance of 0.05 significance?

2013-01-06T14:31:00.001-08:00

Why do we tend to use a statistical significance level of 0.05? When I teach statistics or mentor colleagues brushing up, I often get the sense that a statistical significance level of α = 0.05 is viewed as some hard and fast threshold, a publishable / not publishable step function. I've seen grad students finish up an empirical experiment and groan to find that p = 0.052. Depressed, they head for the pub. I've seen the same grad students extend their experiment just long enough for statistical variation to swing in their favor to obtain p = 0.049. Happy, they head for the pub.

Clearly, 0.05 is not the only significance level used. 0.1, 0.01 and some smaller values are common too. This is partly related to field. In my experience, the ecological literature and other fields that are often plagued by small sample sizes are more likely to use 0.1. Engineering and manufacturing where larger samples are easier to obtain tend to use 0.01. Most people in most fields, however, use 0.05. It is indeed the default value in most statistical software applications.

This "standard" 0.05 level is typically associated with Sir R. A. Fisher, a brilliant biologist and statistician that pioneered many areas of statistics, including ANOVA and experimental design. However, the true origins make for a much richer story.

Let's start, however, with Fisher's contribution. In Statistical Methods for Research Workers (1925), he states

The value for which P=0.05, or 1 in 20, is 1.96 or nearly 2; it is convenient to take this point as a limit in judging whether a deviation ought to be considered significant or not. Deviations exceeding twice the standard deviation are thus formally regarded as significant. Using this criterion we should be led to follow up a false indication only once in 22 trials, even if the statistics were the only guide available. Small effects will still escape notice if the data are insufficiently numerous to bring them out, but no lowering of the standard of significance would meet this difficulty.

The next year he states, somewhat loosely,

... it is convenient to draw the line at about the level at which we can say: "Either there is something in the treatment, or a coincidence has occurred such as does not occur more than once in twenty trials."...

If one in twenty does not seem high enough odds, we may, if we prefer it, draw the line at one in fifty (the 2 per cent point), or one in a hundred (the 1 per cent point). Personally, the writer prefers to set a low standard of significance at the 5 per cent point, and ignore entirely all results which fail to reach this level. A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.

(See http://www.jerrydallal.com/LHSP/p05.htm)

And there you have it. With no theoretical justification, these few sentences drove the standard significance level that we use to this day.

Fisher was not the first to think about this but he was the first to reframe it as a probability in this manner and the first to state this 0.05 value explicitly.

Those two z-values in the first quote, however, hint at a longer history and basis of the different significance levels that we know and love. Cowles & Davis (1982) On the Origins of the .05 level of statistical significance describe a fascinating extended history which reads like a Whos Whos of statistical luminaries: De Moivre, Pearson, Gossett (Student), Laplace, Gauss and others.

Our story really begins in 1818 with Bessel who coined the term "probable error" (well, at least the equivalent in German). Probable error is the semi-interquartle range. That is, ±1PE contains the central 50% of values and is roughly 2/3 of a standard deviation. So, for a uniform distribution ±2PE contains all values but for a standard normal it contains only the central 82% of values. Finally, and crucially to our story,

±3PE contains the central ~95% of values. 1 - 0.95 = 0.05
People like Quetelet and Galton had tended to express variation or errors outside some typical range in terms of ±3PE, even after Pearson coined the term standard deviation.

There you have the basis of 0.05 significance: ±3PE was in common use in the late 1890s and this translates to 0.05. 1 in 20 is easier to interpret for most people than a z value of 2 or in terms of PE (Cowles & Davis, 1982) and thus explains why 0.05 became more popular.

In one paper from the 1890s, Pearson remarks on different p-values obtained as

p = 0.5586 --- "thus we may consider the fit remarkably good"

p = 0.28 --- "fairly represented"

p = 0.1 --- "not very improbable that the observed frequencies are compatible with a random sampling"

p = 0.01 --- "this very improbable result"

and here we see the start of different significance levels. 0.1 is a little probable and 0.01 very improbable. 0.05 rests between the two.

Despite this, ±3PE continued to be used as the primary criterion up to the 1920s and is still used in some fields today, especially in physics. It was Fisher that rounded off the probability to 0.05 which in turn, switched from a clean ±2σ to ±1.96σ.

In summary, ±3PE --> ±2σ --> ±1.96σ --> α = 0.05 more accurately describes the evolution of statistical significance.

Howto: build a news aggregator (in 100 LOC)

2012-12-31T12:13:00.001-08:00

In this post, I'll cover how to build a crude, simple but working news aggregator in about 100 lines of python (source code). While a complete news aggregator will comprise of at least three steps:

filter: select a subset of stories of interest, i.e. the top news -- e.g., that Zynga's shuttering of several games will be one of the most interesting to Techmeme's readership in the screenshot below
aggregate: cluster like documents --- that Techcrunch and GamesIndustry International are both covering the same story
choose: choose the best representative story -- that the Techcrunch story should be highlighted over the GamesIndustry International's story,

in this article, we will focus on the second step only, aggregation of similar articles.

Let us state the goal explicitly: given multiple sources of documents, such as RSS news feeds, cluster together similar documents that cover the same material. For instance, both Reuters and Associated Press might cover Hillary Clinton's admission to hospital (which was a major story at time of writing). In the google news screen shot below we can see that USA Today, Detroit Free Press, Boston Herald and at least 10 other news organizations covered that story. We wish to recognize that these are the same underlying story and cluster or aggregate them.

My approach was the following:

parse RSS feeds
extract keywords using TF-IDF (i.e., term frequency * inverse document frequency)
union those to a set of features that encompasses the whole corpus
create a feature vector for each document
compute matrix of cosine similarities
run hierarchical clustering and plot dendrogram
extract final clusters of interest

I'm not claiming this is an optimal scheme or will necessarily scale well and I did not focus on vectorized operations. It does, however, produce non-embarassing results in 100 lines of code. Later, we will discuss other approaches including the algorithm versus human debate.

Let's build an aggregator. To start, we will need some data sources. I chose a handful of RSS feeds about technology and IT news:

feeds = [

'http://www.sfgate.com/rss/feed/Tech-News-449.php',

'http://feeds.feedburner.com/TechCrunch/startups',

'http://news.cnet.com/8300-1001_3-92.xml',

'http://www.zdnet.com/news/rss.xml',

'http://www.computerweekly.com/rss/Latest-IT-news.xml',

'http://feeds.reuters.com/reuters/technologyNews',

'http://www.tweaktown.com/news-feed/'

]

(We can assume that these source have already mostly accomplished the first news aggregator step: filtering.)

Next, we need to parse out the source. I decided to use the document title and description which is a shortish summary. I used the feedparser python library and the natural language processing library NLTK. For each document, I concatenate the title and description, convert to lowercase, and remove one character words. I store those final documents in a corpus and also store the titles to make the final results more interpretable:

import feedparser

import nltk

corpus = []

titles=[]

ct = -1

for feed in feeds:

d = feedparser.parse(feed)

for e in d['entries']:

words = nltk.wordpunct_tokenize(nltk.clean_html(e['description']))

words.extend(nltk.wordpunct_tokenize(e['title']))

lowerwords=[x.lower() for x in words if len(x) > 1]

ct += 1

print ct, "TITLE",e['title']

corpus.append(lowerwords)

titles.append(e['title'])

Here is a sample of those data (titles only):

0 TITLE Tweeters say the dumbest things

1 TITLE IFixit thrives by breaking tech items down

2 TITLE Earthcam's New Year's Live 2013

3 TITLE Content Fleet offers publishers hot tips

...

96 TITLE Sony stops shipments of PS2 console in Japan

97 TITLE RIM's first patent payment to Nokia: $65 million

98 TITLE ZTE aiming for thinner 5-inch 1080p-capable smartphone, Grand S has grand dreams

99 TITLE Japanese security firm to take to the sky in 2014, will unleash crime-fighting drones

We will extract keywords using TF-IDF. I modified the following from this post:

import math

from operator import itemgetter

def freq(word, document): return document.count(word)

def wordCount(document): return len(document)

def numDocsContaining(word,documentList):

count = 0

for document in documentList:

if freq(word,document) > 0:

count += 1

return count

def tf(word, document): return (freq(word,document) / float(wordCount(document)))

def idf(word, documentList): return math.log(len(documentList) / numDocsContaining(word,documentList))

def tfidf(word, document, documentList): return (tf(word,document) * idf(word,documentList))

I only want to extract a few top keywords from each document and I'll store the set of those keywords, across all documents, in key_word_list:

import operator

def top_keywords(n,doc,corpus):

d = {}

for word in set(doc):

d[word] = tfidf(word,doc,corpus)

sorted_d = sorted(d.iteritems(), key=operator.itemgetter(1))

sorted_d.reverse()

return [w[0] for w in sorted_d[:n]]

key_word_list=set()

nkeywords=4

[[key_word_list.add(x) for x in top_keywords(nkeywords,doc,corpus)] for doc in corpus]

ct=-1

for doc in corpus:

ct+=1

print ct,"KEYWORDS"," ".join(top_keywords(nkeywords,doc,corpus))

Here is a sample of those data:

0 KEYWORDS she tweeted athletes south

1 KEYWORDS ifixit repair devincenzi items

2 KEYWORDS earthcam live famed sound

3 KEYWORDS fleet content tips publishers

...

96 KEYWORDS playstation sony console ouya

97 KEYWORDS payment nokia patent agreement

98 KEYWORDS zte grand boots ces

99 KEYWORDS drones soon 2014 drone

Now that we have this superset of keywords, we need to go through each document again and compute TF-IDF for each term. Thus, this will be likely be a sparse vector as most of the entries will be zero.

feature_vectors=[]

n=len(corpus)

for document in corpus:

vec=[]

[vec.append(tfidf(word, document, corpus) if word in document else 0) for word in key_word_list]

feature_vectors.append(vec)

With a vector of TF-IDF for each document, we can compare the similarity with each other. I decided to use cosine similarity (and yes, this matrix is symmetric and thus the code is far from optimized):

import numpy

mat = numpy.empty((n, n))

for i in xrange(0,n):

for j in xrange(0,n):

mat[i][j] = nltk.cluster.util.cosine_distance(feature_vectors[i],feature_vectors[j])

We now have a single similarity number for each pair of documents.

The next step is clustering. Initially, the most obvious might be to use k-means or k-means++ clustering. It is simple, well-known and can work relatively well. As well as getting stuck on local optima, the major problem here, however, is that one has to choose k. This is not always easy. Moreover, I strongly suspect that the characteristics of news changes dramatically. That is, on presidential election day, almost all top stories will be about voting and speculation on the results. That is, there may be very few, very large clusters. On slow news days that story landscape may be very different. As such, I think it would be difficult to choose a constant k across days and the characteristics of the data may change so much that a scheme to choose k dynamically might be tricky.

I chose a different approach: agglomerative or hierarchical clustering. Here clusters are grown by fusing neighboring documents to form a tree. The structure of that tree may change radically among different days but we can choose a similarity threshold to prune the tree to a set of final clusters. That similarity metric can be constant over days.

I used the hcluster python library and saved the dendrogram to file:

from hcluster import linkage, dendrogram

t = 0.8

Z = linkage(mat, 'single')

dendrogram(Z, color_threshold=t)

import pylab

pylab.savefig( "hcluster.png" ,dpi=800)

Finally, we need to chose a threshold from which to prune -- how similar is similar enough. 0.8 worked best for my data. There is then a little work to extract the clustered items from the dendrogram and print out the IDs and titles of the final clusters.

def extract_clusters(Z,threshold,n):

clusters={}

ct=n

for row in Z:

if row[2] < threshold:

n1=int(row[0])

n2=int(row[1])

if n1 >= n:

l1=clusters[n1]

del(clusters[n1])

else:

l1= [n1]

if n2 >= n:

l2=clusters[n2]

del(clusters[n2])

else:

l2= [n2]

l1.extend(l2)

clusters[ct] = l1

ct += 1

else:

return clusters

clusters = extract_clusters(Z,t,n)

for key in clusters:

print "============================================="

for id in clusters[key]:

print id,titles[id]

Here then, in combination with the dendrogram above, are my final results:

=============================================

35 Huawei linked to plan to sell restricted equipment to Iran

65 Exclusive: Huawei partner offered embargoed HP gear to Iran

=============================================

10 UK's Pearson invests in Barnes & Noble's Nook

73 Pearson to buy stake in Nook, Barnes & Noble shares up

=============================================

21 Kim Dotcom To Host Mega’s Launch Event At His New Mega Zealand Mansion Next Month

94 Kim Dotcom to host Mega's launch event at his New Zealand mega mansion next month

=============================================

90 Data suggests Facebook's Poke has led to online buzz around Snapchat

93 Snapchat videos and photos can be saved after all

=============================================

14 Apple CEO Tim Cook paid $4.16 million

41 Apple's Tim Cook sees his 2012 pay fall 99 percent

=============================================

8 Netflix CEO gets pay bump after 2012 cut

72 Netflix increases CEO Hastings' 2013 salary to $4 million

=============================================

95 Nissan Leaf owners can get their batteries refreshed under new warranty options

99 Japanese security firm to take to the sky in 2014, will unleash crime-fighting drones

=============================================

83 Windows RT ported to HTC HD2

77 Samsung Galaxy S III extended battery arriving in Germany on January 5

92 RumorTT: Samsung Galaxy S IV rumored for April Launch with integrated S Pen

75 Unnamed BlackBerry 10 device slides through FCC with AT&T-capable LTE abilities

98 ZTE aiming for thinner 5-inch 1080p-capable smartphone, Grand S has grand dreams

=============================================

40 HP confirms: Feds investigating the Autonomy acquisition

52 US Department of Justice is investigating allegations of accounting fraud at Autonomy

11 HP says gov't investigating troubled Autonomy unit

39 Autonomy founder fires back at HP after news of DOJ inquiry

=============================================

55 Top 10 broadband stories of 2012

63 BT wins BDUK broadband contract in Norfolk

=============================================

97 RIM's first patent payment to Nokia: $65 million

84 Fujitsu blames weak Windows 8 demand for poor sales

87 Cheezburger network raises $5M in funding for LOLcats, FAIL Blog, others

79 Samsung has big plans for Silicon Valley with 1.1 million square foot R&D center

89 Apple drops patent claim against Samsung Galaxy S III Mini

The complete output is here.

To sort the items within a cluster, the final news aggragator step, one could sort by PageRank using code such as this.

We see good cluster around Huawei, Person, Kim Dotcom, Facebook/Snapchat, Tim Cook, and Netflix. All of those seem reasonable. The next is spurious. Next is a broader cluster around smartphones. The next is a good four story cluster around the HP and Autonomy scandal. The next cluster relates to broadband and the documents are related but not really about the same material. Finally, there is a cluster around "millions" which cover both dollars and square foot.

The results are not great. I'm not going to build a techmeme.com competitor from this. However, they are surprisingly respectable for a first pass and for so few lines of code.

To try to improve results, I tried doing a weighted TF-IDF. That is, I suspect that words in the title are relatively more important than those in the description. Thus, I set a weight 1 for each word in the description and weight title_wt > 1 for each word in the title. When we compute term frequency I sum the weights for the words present rather than counts. The code for this change is here.

I did not find that this improved results. The dominant parameter by far was the number of keywords from each document (nkeywords=4 above). Increasing this dramatically increases false positives (i.e. larger mixed clusters, such as the million cluster above). Reducing it below 4 led to very few clusters (for a given document similarity threshold). Actually, I was surprised that the optimal value of 4 was so low. I had expected a larger number would have been needed to disambiguate different the articles.

A potential improvement that I have not tried is stemming the words. In the sample keyword output above you might have noticed the following:

99 KEYWORDS drones soon 2014 drone

Drones and drone are the same noun. Had I stemmed these it would have increased the TF-IDF of drone and created a space for an additional keyword.

I may be able to tweak my current code and improve results. I could try different parameters, try different similarity metrics, different clustering schemes but it is not clear how good the results could be. Could I match a human curator? According to Gabe Rivera, the founder of Techmeme.com, algorithms are not currently a match for humans. His site employs a small team of editors [http://techmeme.com/about] to curate the top news:

Our experience leads us to believe that a thoughtful combination of both algorithmic and human editing offers the best means for curating in a space as broad as technology.

I somehow started thinking about news aggregators just a few days before prismatic got a large round of funding. In the days that followed there was some debate about algorithms versus humans. Here is Techcrunch:

News aggregation is a glory-less pursuit, where even the companies that do succeed have to grapple with the fact that no algorithm is ever going to be as good as the human brain at judging what news/content will appeal to said brain

And Gabe Rivera (from Techmeme.com):

"A lot of people who think they can go all the way with the automated approach fail to realize a news story has become obsolete," said Rivera, explaining that an article can be quickly superseded even if it receives a million links or tweets. from gigaom

and paraphrasing:

Some decisions, after all, are best left to humans. When a news story breaks, for example, or a much-hyped product launches, a wealth of news coverage follows. Which is the best version? While Techmeme’s algorithms often derive this answer from “signals” on the Internet, sometimes surfacing the right story for the audience — perhaps the most comprehensive take or the most nuanced coverage — requires human intervention. outbrain.com

and finally in 2008, at the time Rivera introduced human editors

I should note that the experience of introducing direct editing has been a revelation even for us, despite the fact that we planned it. Interacting directly with an automated news engine makes it clear that the human+algorithm combo can curate news far more effectively that the individual human or algorithmic parts. It really feels like the age of the news cyborg has arrived. Our goal is apply this new capability to producing the clearest and most useful tech news overview available

(See also this thread.)

As mentioned earlier, news aggregation is not just about clustering similar stories. There are really three steps: story selection, clustering like documents, and selecting best item from each cluster. This is certainly a set of hard problems. Which stories might be of interest depends on the audience and other contexts. Clustering similar documents is probably the easiest subtask and that is fraught with problems, e.g. that a bag-of-words model does not distinguish millions of dollars versus millions of square feet. (Thus, a n-gram model might be another possible approach.) There is a sea of potential news sources, they have different foci, reputations come and go, quality changes, news organization editors vary and change too. As Gabe mentions, there is a temporal aspect too where there is a window of interest before the news get stale.

There is no reason that an algorithm or suite of algorithms could not do this well. There are certainly a lot of social cues now that an algorithm could use to identify trending, interesting stories: the number of likes, shares or retweets; the reputation of those sharing the news; the number or rate of comments on those stories etc. To understand the audience and refine itself, an algorithm could use direct user feedback and again shares / like / retweets of those top stories. However, it remains a hard problem. With the big news sources covering core issues, smaller news operations have to work a different angle and sometimes those are the most interesting. Journalists are not consistent: they might write a blazing article one day and a mediocre the next. Thus, for now the algorithmic base + human editors or a pure crowd-sourced filtering (such as reddit) do seem to produce the best quality results.

Detecting credible tweets

2012-12-27T08:14:00.000-08:00

There has been some buzz in the last week or so around a paper detecting the credibility of tweets. As Twitter has grown, so has spam. Moreover, during crises such as tsunamis, the Arab spring, the UK riots and so on, one sees deliberate strategies to spread disinformation. The effects of this can range from inconvenience, to wasted time and resources to being downright dangerous. Thus, a system to filter out such disinformation would be valuable for media, first responders and others.

In Credibility Ranking of Tweets during High Impact Events, PSOM 2012, Gupta and Kamaguru attempted to identify features that helped identify credible information versus opinion versus spam. The built a supervised learning model using features of the tweets and the senders and ran those through a regression model and re-ranking scheme that they claim provided a reasonable predictive ability.

To do this, they collected 35 million tweets from 6 million users based on search terms from trending topics during summer 2011. These were high impact events meaning they produced at least 25k tweets and persisted as a trending topic for more than 48 hours. They then selected 14 trending topics---including UK riots, Hurricane Irene, Steve Jobs resignation and the bomb blast in Mumbai---and annotated the associated tweets to flag whether each tweet was credible. In short, a human had to decide whether each tweet was definitely credible / seems credible / definitely incredible / I can't decide and whether it contained information or not, was unrelated to the news event, or was spam. An example of the three classes is shown below.

They found that 30% of tweets provided information, 17% was credible and 14% was spam. They then proceeded to extract a large set of features from the tweets, the sender and their user profile. These features included:

Tweet: number of words, number of characters, whether it was a retweet, whether it contained a URL, whether it contained emoticons, number of hashtags etc.

Sender: age, number of followers, number of friends, description length, whether it was a verified account etc.

They performed a logistic regression (1 = credible, 0 = not credible/spam/opinion) and found the following significant indicators (p < 0.001):

number of characters and unique characters present in tweet --- likely tweets with hashtags, @mentions and URLs contain more unique characters and so are more informative
presence of swear words --- indicates opinion
inclusion of pronouns and presence of sad / happy emoticons --- fact based tweets are less personal
p<0.01: presence of URL --- often linked to images, video and resources related to the event.

They found that "low number of happy emoticons [:-), :)] and high number of sad emoticons [:-(, :(] act as strong predictors of credibility". Now, most of the high impact (arguably 13/14) events in their list are negative so one could imagine that for a general model that covered both positive and negative events, this feature would be use of emoticons: happy emoticons for happy events, sad emoticons for sad events. However, one could also argue that use of happy emoticons such as :-) could mean a joke, that the comment is not credible. We will see data from a second study below that finds the same results so it is more likely to be the latter: sad emoticons only are strongly linked to credible information.

They also used SVM and pseudo relevance feedback (PRF) to improve results. They ranked documents using RankSVM and using a unigram model identified the top K tweets and reranked them using a BM25 similarity metric. (This metric is a function of TF, IDF, K, tweet length, and some constants. See paper for full description.) To evaluate the results they used the Normalized Discounted Cumulative Gain metric. For the top 25 tweets, they achieved 0.37 NDCG, a statistically significant (p < 0.05) improvement over non ranked data. Unfortunately, there is scant further details about their analysis. I have no idea as to the accuracy, precision, recall or F-scores of their classifier. Saying that an additional layer of analysis improved results is not as useful as detailing fully how good the initial results were.

Gupta and Kamaguru are not the only ones working in this field. Castillo et al. (2011) published a more compelling paper entitled "Information Credibility on Twitter" with a more detailed statistical analysis. They too sucked up tweets from trending topics, this time over a 2 month period. They used Mechanical Turk to annotate the newsworthiness and credibility and asked the evaluators to write a sentence justifying their assessment. The extracted a sets of features, similar but larger and more detailed than above.

They tried a number of learning schemes including SVM, decision trees, decision rules, and Bayes networks. Results across these techniques were comparable with the best coming from a J48 decision tree. Their J48 classifier achieves an accuracy equal to 89%. For detecting newsworthy tweets they achieved an impressive F1 score of 0.924. For predicting credibility they also found that a J48 tree worked best and found the following significant features: presence of questions mark (non credible), sentiment (positive: no credible, negative: credible), presence of negative emoticons (credible). More active users are more credible (I wonder if this is related to professional media organizations such as CNN, Reuters, local news organizatins whose business it is to be both credible and active.) Together, these feature produced a model with 86% accuracy. See table below.

In both these studies, the features make sense. Active users, tweeting frequently with lots of followers, linking to relevant material and who have a reasonably fleshed out user profile. With 86% accuracy from a balanced training set, such features and associated model could form the basis of a reasonably good filter system for a twitter stream and in the case of emergency services could save valuable resources and possibly lives.

P.S. it is worth checking out the truthiness system from http://truthy.indiana.edu/

On the unreasonable effectiveness of data: why are more data better?

2012-12-14T17:42:00.000-08:00

Update 11/30/14: I incorrectly named the authors of the "unreasonable effectiveness study, now corrected.

In the paper "The unreasonable effectiveness of data" [Halevy et al, 2009], Halevy, Norvig and Pererira, all from Google, argue that interesting things happen when corpora get to web scale:

"simple models and a lot of data trump more elaborate models based on less data".

In that paper and the more detailed tech talk given by Norvig, they demonstrate that when corpora get to hundreds of millions or trillions of training sample or words, very simple models with basic independence assumptions can outperform more complex models such as those based on carefully-crafted ontologies with smaller data. However, they provided relatively little explanation as to why more data are better. In this post, I want to attempt to think through that.

I propose that there are several classes of problems and reasons for why more data are better.

The first are nearest neighbor type problems. Halevy et al. mention "James Hay and Alexei A. Efros addressed the task of scene completion: removing an unwanted, unsightly automobile or ex-spouse from a photograph and filling in the background with pixels taken from a large corpus of other photos." [Hays and Efros, 2007 ]

Norvig presented the following schematic:

and described it as a "data threshold" in which results went from really bad to really good.

I'm not convinced that there is any threshold or anything that resembles a phase transition. This seems to me to be a problem of finding the closest match. The more data, the closer the expected match.

Hays and Efros (2007) mention:

"Indeed, our initial experiments with the gist descriptor on a dataset of ten thousand images were very discouraging. However, increasing the image collection to two million yielded a qualitative leap in performance... Independently, Torralba et al. [2007] have observed a similar effect with a dataset of up to 70 million tiny (32x32) images...It takes a large amount of data for our method to succeed. We saw dramatic improvement when moving from ten thousand to two million images" [Hays and Efros, 2007]

There is a large difference in those corpus sizes and a "qualitative leap" is not the same as a threshold (sensu phase transition).

More data can dramatically affect the metrics from simple effects. For instance, consider a sample of size n from a standard normal. How does the minimum of that sample vary with n? Let's create samples of different sizes and plot the minimum using the following R code

x<-seq(1,7,0.5)

y<-vector(mode="numeric",length=length(x))

for (i in 1:length(x)){ y[i] <- min(rnorm(10^(x[i]))) }

plot(x,y,xlab="Sample size, n (log10 scale)",ylab="Minimum value of sample",type="b")

The minimum decreases loglinearly. This is a case of extrema from an unbounded tail. Perhaps more relevantly, here, for a minimization problem such as scene matching there is a lower bound: for all intents and purposes, a perfect match. For instance, perhaps someone else stood in the same exact tourist spot and took a picture of the exact same scene but without the obstructing car.

I think this is what is happening in Norvig's schematic. At a certain corpus size we've found a really good match and a larger corpus doesn't and cannot improve results.

In summary, for nearest neighbor type minimization problems with a non-negative distance function (meaning that the cost function has a lower bound of zero), that distance function will, on average, decrease monotonically with data or sample size.

The second class are counting or relative frequency problems. These were the primary focus of Halevy et al. Norvig's presented a few examples. In segmentation, the task is to split a string such as "cheapdealsandstuff.com" into the most likely sequence of words. These strings are short enough to do a brute force in terms of possible partition, but for each partition we have to assess the likelihood of that partition. The simplest assumption is to assume independence among words. Thus, if Pr(w) is the probability of a word w given some corpus we can compute, say

Pr(che,apdeals,andstuff) = Pr(che) * Pr(apdeals) * Pr(andstuff).

...

Pr(cheap,deals,and,stuff) = Pr(cheap) * Pr(deals) * Pr(and) * Pr(stuff).

One can of course also use n-grams, e.g. for bigrams: Pr("cheap deals") * Pr("and stuff")

A second example that Norvig covered was spell checking. Here we can take a misspelled word and compute the likelihood of the possible variants to suggest the most likely form.

In both cases, we need a corpus that includes both common and uncommon phrases and we need counts of the occurrences of those phrases to compute relative frequency. The larger and more comprehensive the corpus, the better. I think that there are two statistical effects going on here:

the larger the corpus, the better the quality of the estimate of the relative frequency. This is the law of large numbers.
the larger the corpus, the more likely it is to include unusual phrases, i.e. the long tail. This is an unbounded effect. The more of the web is indexed, there will always be new, never-seen-before phrases. The problem is exacerbated in that the distribution of words in the english language is a power law. (e.g. Zipf, G. The Psycho-Biology of Language. Houghton Mifflin, Boston, MA, 1935.). This means that the tail is especially long and thus especially large samples are needed to capture those rare phrases.

A third class are estimating univariate distribution problems. I recently encountered an interesting example while attending a tech talk from Peter Skomoroch from LinkedIn. He showed a plot of likelihood of a member having particular software-related job titles versus number of months since graduation. What we see from the data is that the distributions of "Sr Software engineer" and "senior software engineer" are almost identical, as one would expect as they are synonyms, as are "CTO" and "chief technology officer". This then presents an interesting way of identifying synonyms and so deduping the data rather than maintaing a huge master list of acronyms and abbreviations. This is only possible because of the scale of data that they have where the distribution they can generate is reliable and presumably close to the true underlying population distribution.

A fourth class are general multivariate or correlational problems in which we are trying to estimate the relationship among variables. This could be estimating the relationship y = f(x) or perhaps estimating the joint pdf of many variables. We might use this for word sense disambiguation (e.g. is the document referring to pike the fish or pike the pointed weapon?) or to build up a dossier of associated features or concepts about an entity (e.g. a company has an associated CEO, head office, tax ID and so on). Here we are interested in the correlations among words or phrases. The problem is that web documents are very high dimensional and embarking on high dimensional problems like these we are under the curse of dimensionality, that data become very sparse. Thus, one effect of larger samples is to increase the density of data across state space. Again, with larger samples we can estimate metrics such as location metrics (mean, median and other metrics of the center of distributions) more accurately. We can also estimate joint pdfs more accurately. Below is a simple example from the following code:

par(mfrow=c(1,2))

plot(mvrnorm(100, mu = c(0, 0), Sigma = matrix(c(1, .9, .9, 1), 2)),xlab="X",ylab="Y",ylim=c(-4,4))

title("n = 100")

plot(mvrnorm(10000, mu = c(0, 0), Sigma = matrix(c(1, .9, .9, 1), 2)),xlab="X",ylab="Y",ylim=c(-4,4))

title("n = 10000")

At left is a small sample. It could easily be interpreted as linear. At right, with a larger sample, the true underlying bivariate normal is more obvious. Clearly, this is a trivial example. The point is that for higher dimensions you will much larger sample sizes to estimate the joint pdfs well.

This is clearly a cursory answer as to why more data are better. Quality data are still preferred. However, for many organizations, such as Google, Twitter, LinkedIn and Facebook where content is user-generated, is often free-form text and/or covers many domains (and so deep-cleaning and the use of ontologies is infeasible) then what we see that having very large datasets can compensate for the noise. It all evens out in the end and in the case of nearest neighbor problems, the solution will always be better.

What is a data scientist?

2012-12-03T09:34:00.000-08:00

Over the last year, there has been a lot of discussion about what is data science and who are data scientists. There are those that think that a data scientist is simply a glorified, over-hyped term for a business intelligence analyst. On the other hand, there are those that equate it with big data. I think that neither of those interpretations are correct and here is my take.

I like to think of it in the following manner: imagine a triangle. In one corner are business intelligence (BI) analysts. In a second corner are data engineers and in the third corner are applied statisticians.

Each corner represents an extreme form of those roles. For BI, it might represent someone who knows a little SQL and does everything else in Excel. There is nothing wrong with this at all, Excel can get you far, but my underlying point is that most BI analysts do much more than this, they protude much further into the triangle. The extreme data engineer might be someone who owns some data processing system, is agnostic about the data flowing through their system and cares solely about gathering, throughput, scaling, peak loads, data gathering logging and so on. Again, most data engineers do much more than this. Finally, the extreme applied statistician might be someone involved in statistical modeling or algorithm development. For instance, they might study recommender systems with toy datasets somewhat in a vacuum from real problems. Again, most statisticians have a broader role and set of skills than this.

While BI is mostly in their corner of the triangle, there are some BI analysts that overlap significantly with data engineers. They are involved in data gathering, architectural decisions about data storage, processing, visualization solutions and replication. They may query with big data technologies such as pig, hive and so on. Some BI analysts may also stray significantly towards the data scientist corner. For instance, they may be involved in some sort of predictive modeling, significant tests, experimental design, may use tools such as R and so on.

Data engineers may show some overlap with BI, perhaps implementing tools for data analysis, doing some analyses, care about the meaning and value of the data in their systems. They may also work closely with data scientists. For instance, they may take prototypes and algorithms implemented by the data scientists and refactor them or completely implement them in a new language to scale. They may understand the core algorithms sufficiently well to improve them for faster performance or better results.

Data scientists are a group of people that cover some central swathe of the triangle, the blue area of triangle (which is purely illustrative). Within that we might find specializations: the machine learners, the data miners, the recsys guys. For now, let's label them collectively as data scientists.

Some data scientists may stray towards BI and write retrospective analyses and, yes, may even use Excel. They may also work significantly with big data and work with map reduce, pig, hive, kafka, fluentd etc. and be concerned with robust, scalable, high throughput systems. They may also be concerned with algorithm development, predictive analytics and a while lot more. No single person, however, covers all these areas. They are will cover some proportion of the area. Thus, one data scientist may be represented by the orange blobs of the triangle while a different data scientist may be represented by the red blob.

This then, I think, is the root of some of the problem here. Job ads for data scientists tend to cover the large blue area. They have requirements or nice-to-haves that cover predictive analytics, recommenders, big data, A/B testing and coding. And, certainly when I interview data scientists, I will often touch upon all of these, especially if they make it to an onsite interview. However, I don't expect that any single candidate to be great or have experience in all these areas. If they did, their experience would be so shallow so as to be useless. The trick then is to define the area of the triangle you need to cover with your data science team and then hire staff that collectively, not necessarily individually, cover that area.

What then is a data scientist? My favorite definition comes from Josh Wills:

Data Scientists (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.

While inadequate, this definition does capture the essence of the idea that there is a continuum in terms of skills and experience. They are not at the extremes of the triangle but each person is a bit of this and a bit of that. It also captures the idea that a typical (non-extreme) data scientist must be able to code. It might be in R or matlab but it must be code and not solely equations. More preferable is some scripting language such as ruby or python that allow rapid prototyping, and the ability to tap into libraries such as numpy, scipy and NLTK. If you can't code, you are unlikely to get too far as a data scientist. Some discuss data scientists' roles as those that find the needle in the haystacks or the gold in the hills. To do that, however, is going to require some code to automate processing or to learn parameters of some model.

I think data science is more about building data products and new data-driven features and services. I don't mean shuffling existing data around as web services but generating new data that drives a feature or product that, in turn, drives value to the organization. This might be an internal sales forecast, website personalization or a recommendation. In my role at onekingslane.com, some of my time involves image processing, generating entirely new data from product images which can drive new site features such as faceted search or deeper BI analyses.

Data science can involve big data but not necessarily. If you are Facebook, Twitter, or LinkedIn with a large and very active user base then it is a necessity. But big data is not a necessary condition for data science. In my last last post, we built a very respectable sentiment analyzer with less than a 100Kb training set. Quality data are preferred but if one has big data to offer, we'll take it.

In summary, data scientists vary. They cover a lot of territory, certainly collectively but not necessarily individually. Some are more statistical, some are more analytical and into data visualization, and some are more computational. While they may overlap with BI and data engineers, they are not the same, are complementary, and do possess a certain novel mix of skills and talents that can provide significant value to their organizations.

Introduction to Sentiment Analysis

2012-11-29T22:06:00.000-08:00

In this post we are going to explore sentiment analysis using python. Let us first define the problem. Given a corpus of documents, we want to train a classifier to classify documents into one of several classes:

positive: e.g., "that movie was awesome"

negative: e.g., "that movie sucked"

We may also want a third category:

neutral: e.g., "I saw the movie at the Odeon".

In reality, we may have a fourth category unknown where we cannot tell. For instance, it may contain a document in a foreign language, a set of stop words only, or a set of words that our classifier does not understand, e.g. SMS acronyms (e.g., "cu l8r"). For simplicity, we can simply not classify these documents or we can categorize them as neutral.

To train the classifier we will need to choose a set of features, the facets of data that the classifier will work on. The simplest scenario is to use each word in the document. (Later, we will discuss more complex features and other features.)

This is a supervised learning problem so we will need a labeled corpus of data. That is, a set of documents each of which has been labeled with one of our class names or IDs. One of the simplest ways is a CSV or a TSV file:

pos that movie was awesome

neu I saw the movie at the Odeon

pos I love this place

pos I am happy

....

We will need a learner, an algorithm that will learn to classify. In most examples that I've seen online, people have used a naive Bayes classifier but there are many others that one could choose.

Importantly, sentiment analysis is very context sensitive. If you train a classifier using movie review data, it likely will not fare well classifying documents about election results or your startup's product. A classifier trained on US english tweets may or may not classify UK tweets well.

Finally, we cannot expect to assign sentiment correctly 100% of the time. Even humans can often disagree about the sentiment of documents. There are several reasons:

Cultural differences mentioned above. For example, "rocking" and "sick" are positive adjectives to members of some demographics and cultures but not others.
English is a hard and ambiguous language. For instance, consider Chomsky's example: "old men and women". Is that [old men] and [old women] or does it mean [old men] and [women]. Another example is Eats, Shoots & Leaves.
Sarcasm, innuendo, and double entendres. This is one of the current challenges in NLP. For a fun example take a look at the problem of detecting that's what she said.
That people often use a bag of words model for sentiment analysis, at least as a first pass. That is, we analyze a document as a set of words and not a phrase. Thus, we will miss that the "not" in "not good" negates "good". In general, we will miss double negatives and other qualifiers. I love this illustrative example:

During a lecture the Oxford linguistic philosopher J.L. Austin made the claim that although a double negative in English implies a positive meaning, there is no language in which a double positive implies a negative. To which Morgenbesser responded in a dismissive tone, "Yeah, yeah."

Now we have the background out of the way, let's starting building a concrete example.

First, let's grab a corpus. I'm taking the UMICH SI650 - Sentiment Classification training set from Kaggle.

This is a tab-delimited file with 7086 sentences tagged as 1 or 0.

head training.txt

1 The Da Vinci Code book is just awesome.

1 this was the first clive cussler i've ever read, but even books like Relic, and Da Vinci code were more plausible than this.

1 i liked the Da Vinci Code a lot.

1 I liked the Da Vinci Code but it ultimatly didn't seem to hold it's own.

1 that's not even an exaggeration ) and at midnight we went to Wal-Mart to buy the Da Vinci Code, which is amazing of course.

1 I loved the Da Vinci Code, but now I want something better and different!..

1 i thought da vinci code was great, same with kite runner.

1 The Da Vinci Code is actually a good movie...

1 I thought the Da Vinci Code was a pretty good book.

There are many duplicates. Let's remove them:

cat training.txt | sort | uniq > uniq_training.txt

How many positive and negative samples remain?

cat uniq_training.txt | grep ^1 | wc -l

772

cat uniq_training.txt | grep ^0 | wc -l

639

We need to extract features from a document. We'll take unique, lowercase words with more than two characters:

def extract_features(document):

features={}

for word in set(document.split()):

if len(word) > 2:

features['contains(%s)' % word.lower()] = True

return features

>>> extract_features('" A couple of very liberal people I know thought Brokeback Mountain was " stupid exploitation.')

{'contains(very)': True, 'contains(people)': True, 'contains(couple)': True, 'contains(mountain)': True, 'contains(was)': True, 'contains(brokeback)': True, 'contains(liberal)': True, 'contains(exploitation.)': True, 'contains(know)': True, 'contains(thought)': True, 'contains(stupid)': True}

We need to read in our documents:

documents=[]

f = open("uniq_training.txt","r")

for document in f.readlines():

parts= document.strip().split("\t")

documents.append((parts[1],bool(int(parts[0]))))

>>> documents[0]

('" A couple of very liberal people I know thought Brokeback Mountain was " stupid exploitation.', True)

Now extract features from each document in our corpus and split into a training set (80%) and a test set (20%):

import random

random.seed(1234) #so that you can reproduce my results if you wish

random.shuffle(documents)

import nltk

n_train = int(0.8*len(documents))

training_set = nltk.classify.apply_features(extract_features,documents[:n_train])

test_set = nltk.classify.apply_features(extract_features,documents[n_train:])

>>> training_set[0]

({'contains(very)': True, 'contains(people)': True, 'contains(couple)': True, 'contains(mountain)': True, 'contains(was)': True, 'contains(brokeback)': True, 'contains(liberal)': True, 'contains(exploitation.)': True, 'contains(know)': True, 'contains(thought)': True, 'contains(stupid)': True}, True)

Finally, let's now train our classifier:

>>> classifier = nltk.NaiveBayesClassifier.train(training_set)

>>> nltk.classify.accuracy(classifier, test_set)

0.911660777385159

Whoah, 91% accuracy isn't bad at all given that we had a fairly balanced training set:

ct_pos=0

for d in training_set:

if d[1]==True: ct_pos+=1

print ct_pos, len(training_set)-ct_pos

617 511

By that I mean, I would have been suspicious if we had a very imbalanced training set and say 95% of samples were positive.

The proper way to assess the performance is to examine precision, recall and F-scores:

import collections
refsets = collections.defaultdict(set)
testsets = collections.defaultdict(set)
for i, (feats, label) in enumerate(test_set):
refsets[label].add(i)
observed = classifier.classify(feats)
testsets[observed].add(i)

>>> print 'pos precision: %2.3f' % nltk.metrics.precision(refsets[True], testsets[True])

pos precision: 0.951

>>> print 'pos recall: %2.3f' % nltk.metrics.recall(refsets[True], testsets[True])

pos recall: 0.884

>>> print 'pos F-measure: %2.3f' % nltk.metrics.f_measure(refsets[True], testsets[True])

pos F-measure: 0.916

>>> print 'neg precision: %2.3f' % nltk.metrics.precision(refsets[False], testsets[False])

neg precision: 0.871

>>> print 'neg recall: %2.3f' % nltk.metrics.recall(refsets[False], testsets[False])

neg recall: 0.945

>>> print 'neg F-measure: %2.3f' % nltk.metrics.f_measure(refsets[False], testsets[False])

neg F-measure: 0.906

and now the confusion matrix:

observed=[]

actual=[]

for i, (feats, label) in enumerate(test_set):

actual.append(label)

observed.append(classifier.classify(feats))

print nltk.ConfusionMatrix(actual,observed)

| F |

| a T |

| l r |

| s u |

| e e |

------+---------+

False |<121> 7 |

True | 18<137>|

------+---------+

(row = reference; col = test)

These are all great numbers.

Now we can use it:

>>> classifier.classify(extract_features("that movie was awful"))

False

>>> classifier.classify(extract_features("that movie was great"))

True

>>> classifier.show_most_informative_features(16)

Most Informative Features

contains(awesome) = True True : False = 37.8 : 1.0

contains(sucked) = True False : True = 31.0 : 1.0

contains(hate) = True False : True = 24.0 : 1.0

contains(love) = True True : False = 17.8 : 1.0

contains(heard) = True False : True = 10.1 : 1.0

contains(kinda) = True False : True = 8.4 : 1.0

contains(it,) = True False : True = 7.6 : 1.0

contains(want) = True True : False = 6.9 : 1.0

contains(evil) = True False : True = 5.6 : 1.0

contains(those) = True False : True = 5.2 : 1.0

contains(didn't) = True False : True = 5.2 : 1.0

contains(has) = True False : True = 5.2 : 1.0

contains(think) = True False : True = 4.7 : 1.0

contains(miss) = True True : False = 4.7 : 1.0

contains(watch) = True False : True = 4.6 : 1.0

contains(liked) = True True : False = 4.5 : 1.0

>>>

These terms make a lot of sense: awesome, suck, hate, love...

For a more detailed tutorial using NLTK see offical docs. See also this post.

We trained our classifier to learn keywords that have particularly positive or negative associations (the most informative features). Once it has learned that list it is then easy to apply to a new input document. There are list of these learned words available such as AFINN and pattern. For these then, running sentiment analysis is trivial.

In pattern, "the sentiment() function returns a (polarity, subjectivity)-tuple for the given sentence (based on the adjectives in it), with polarity between -1.0 and 1.0 and subjectivity between 0.0 and 1.0".

Let's fire up a sentiment analyzer and use it

>>> from pattern.en import sentiment

>>> print sentiment("that movie was awesome")

(1.0, 1.0)

>>> print sentiment("that movie sucked")

(0, 0)

>>> print sentiment("that movie was ok")

(0.5, 0.5)

>>> print sentiment("that movie was fairly good")

(0.6, 0.8500000000000001)

That's all there is to it.

You can see that returning a continuous value from (-1,1) means that we could easily define neutral documents as those with some intermediate value, say (-0.1,0.1).

Let's return to the issue mentioned earlier that this is just a bag of words model and the issue "not good" versus "good". One thing we can do is to include bigrams in our features. That is, adjacent pairs of words:

>>> from nltk import bigrams

>>> bigrams("That movie was awful".split())

[('That', 'movie'), ('movie', 'was'), ('was', 'awful')]

Let's add bigrams to our extract features function:

def extract_features(document):

features={}

for word in set(document.split()):

if len(word) > 2:

features['contains(%s)' % word.lower()] = True

for bigram in bigrams(document.split()):

features['contains(%s)' % "_".join(i.lower() for i in bigram)] = True

return features

Retraining our classifier with this, we get 93.6% accuracy and a more interesting feature list: I hate, I love, I think, love the...

>>> classifier.show_most_informative_features(16)

Most Informative Features

contains(awesome) = True True : False = 37.8 : 1.0

contains(sucked) = True False : True = 31.0 : 1.0

contains(hate) = True False : True = 24.0 : 1.0

contains(love_the) = True True : False = 22.4 : 1.0

contains(i_hate) = True False : True = 22.2 : 1.0

contains(i_love) = True True : False = 22.1 : 1.0

contains(love) = True True : False = 17.8 : 1.0

contains(code_was) = True True : False = 15.7 : 1.0

contains(i_think) = True False : True = 14.9 : 1.0

contains(3_was) = True True : False = 13.0 : 1.0

contains(i_like) = True True : False = 11.9 : 1.0

contains(like_harry) = True True : False = 10.2 : 1.0

contains(heard) = True False : True = 10.1 : 1.0

contains(i_heard) = True False : True = 9.3 : 1.0

contains(the_mission) = True True : False = 8.6 : 1.0

contains(kinda) = True False : True = 8.4 : 1.0

Hopefully you can see that you can get some good sentiment results with just a few tens of lines of python, if you have the right libraries.

Here is the source code: p-value.info github