Commercial Intent Detection Tool Review

Mary Bowling | 10 May 2008 |

If you're new here, you may want to subscribe to our RSS feed. Thanks for visiting!

Anyone who’s buying or selling advertising online doesn’t need a detailed explanation of why we want to be able to interpret whether a particular keyword phrase indicates an intention to make a purchase or not. We don’t just want traffic to our websites, we want traffic that converts.

While it’s sometimes fairly obvious - either from the words in the search phrase (where can I rent a beach house in Wildwood?) or from the detail within the search phrase (3 bedroom beach house in Wildwood with WiFi) - this is something that optimizers and pay per click campaigners struggle with continually.

The Tool

So, when MSN adCenter Labs unveiled their Commercial Intent Detection Tool about a year ago, it seemed like a great idea. Here’s what they say it does:

Predict a customer’s intention to buy, sell, or complete another kind of transactionbased on her search queries or recently visited URLs.

They backed the release up with an explanation entitled Algorithmic Commercial Intent Detection of Search Queries. The one-pager begins with:

Using algorithmic commercial intent detection, 52% of queries were determined to be associated with commercial intent. The alogorithm was deemed very accurate, with a rate of 90% for detecting both commercial and non-commercial intent.

The Research Behind the Tool

Then, in the manner of content that appears to be scientific, it dryly goes on to explain why and how the algorithm testing was done. Human evaluators were used to check and verify the accuracy of the results. How? Apparently, they were asked if a phrase had commercial intent of not and their answers were correlated with the algo’s results.

We aren’t given any idea of how these human evaluators were chosen or why. Was there something that made these particular humans any more well-equipped to make that determination than others? Do they have an specific expertise in evaluating commercial intent? Did the testing involve 2 testers or 20,000? MSN’s explanation of the study doesn’t say.

So, I dug a little deeper and read the original research paper on Commercial Intent Detection presented by MSN engineers in 2006. I found that the study was based on the opinions of just 3 human evaluators:

We asked 3 human labelers ro label the search queries and pages. Each query or page is labeled as “commercial”. “non-commercial”, or “confused”. Each query was labeled by the 3 labelers separately. After labeling process on queries, we keep the queries/pages that were agreed by at least two labelers with non-confused labels.

What????? Are they kidding? I’m no scientist, but this seems like a very flawed process. Was any of the data cross-checked with the conversion tracking systems of real websites to confirm whether searches for those terms actually resulted in a sale or not?

My Simple Test

Here are the results of a quick test I did using the tool. 1.0 is a 100% probability, so I interpret a .8478 as about 85%. The first column is the search term and the second column is the percentage of probability of commercial intent of the searcher:

I think that pretty much demonstrates the tool’s usefulness - or lack thereof - in evaluating keyword phrases.

How It Could Be Improved

Chances are, we all have at least 3 people on staff who are very well-qualified to label a keyword phrase as having commercial intent or not. Why? Because they regularly view and analyze online sales through sophisticated tracking software. They know the phrases that brought buyers into the websites they manage and the click paths they took once there. Maybe next time, MSN adCenter Labs could at least use experienced Search Engine Marketers as their human labelers and, by using enough of them (way more than 3!), they may get useful data.

Commercial Intent of Web Pages

The second function of the tool is interpreting the commercial intent of a web page. However, there’s no information at all that tells you how MSN adCenter Labs accomplishes that. I tried entering pages with and without ads on them and pages with BUY NOW! all over them and couldn’t really come to any conclusions. This part of the tool doesn’t even have any entertainment value.

Conclusion

The most discouraging thing about MSN’s Commercial Intent Detection Tool is that nothing about it appears to have improved since it was released. It is what it is and it’s not likely to get any better.

I’m afraid I can’t put much stock in this tool as something useful for serious marketers. I think your own experience and instincts can give you much better insight to the commercial intent of a search phrase. Of course, we all have tracker data to fall back on when we really want to interpret commercial intent in a meaningful way.