comScore’s New Panel-Centric Hybrid Measurement

Unfortunately, I miss more than 1/2 of the comScore webinars that are available. Between Alexander, Jamie and I, we try to make as many as we can, but sometimes, I just don’t have the hour to spare. Today, when the reminder for today’s webinar to discuss future changes to the MediaMetrix methodology came up, I wiped my schedule clear. As you may know, comScore provides Canadian (and global) research on website visitors, page views, and demographic profiling. Their data also includes segment indexing, mobile and video consumption and an assortment of planning tools. To date, I have been very happy to have this tool in my back pocket for planning and site audits, but I do know some people in the industry who have been less than impressed about the discrepancies between server side reporting and the data that appears in comScore reports.

Today’s announcement & webinar addressed some of these concerns.

To start with, if you are unfamiliar with comScore’s methodology, you can learn more here. In a nut shell, comScore’s research comes from a panel, just like BBM, Nielsen, PMB and all the other major Canadian research databases. The panelists install a program on their computer(s) that measure their Internet browsing, buying and other activities, in both digital and offline locations. They also try to measure at work consumption, but, due to security and technology barriers, their panel of at work users tends to be considerably smaller. One also needs to take into consideration that of their 2 million panelists globally, 1 million of those panelists are in the states, with Canadian panelists being blended in with the rest of the world. This makes it difficult to magnify smaller portions of the country with any stability, and as such, Alberta, Saskatchewan and Manitoba have been combined into a single entity. While frustrating, this is not the first time that the prairies have been faced with unstable (or non-measured) markets and we work around it as best we can.

The seminar today gave hope to these three provinces as well as any smaller or niche sites that have felt that the comScore data has been misrepresenting their traffic. While panel based research studies are the norm, it’s rare that a local magazine or newspaper will ever have the same number of respondents in a private readership study as compared to a national study such as PMB or NadBank. However, with digital media, each site has an educated IT person who can see exactly how many hits the server is receiving and how many visitors check out their site. The discrepancies between local server side data and comScore national panelist data cause some sites to discredit the data if their comScore reports show them with lower visitor numbers. An additional argument (similar to issues with Nielsen/BBM’s TV viewing panels) is that these panels rarely show an accurate portrayal of visitors to niche sites. Just as a nerdy, cult or vertical-demographic is unlikely to have a Nielsen set box (who is watching conventional TV ANYWAYS these days :) ), they are not getting their news from CanWest or Canoe sites. These users are going to smaller, local sites who are reporting far larger local numbers than comScore ever could. However, server side reporting is not without it’s flaws either.

With real-time reporting (if you are friendly with your IT guys), it’s as easy as popping down to the server room and receiving a 100% accurate count of server request. The only issue is that very few local sites are going to take the time (and resources) to filter out bots, non-humans and other invalid hits. Also, it’s rare that sites have more information about their users beyond their location. A panel-based survey is able to extrapolate demographics and their habits based on collected information. Then, we have the cookie problem.

You may recall my rant following the IABC Mixx Conference this past Spring showing US cookie deletion data. This spurred Alexander to write a post about why cookies are great for the consumer and echoed my concerns about the US data being incongruous with Canadian habits. HURRAH! The day at last has come where I am in possession of the 2008 Canadian cookie deletion numbers.

  • The study spanned 9 months from January – September 2008.
  • They found that 34.3% of Canadians (based on their panel) deleted their cookies once a month with an average of 6 cookies being deleted.
  • 10% of Canadians deleted their cookies once a week with an average of 15 (14.8) cookies on their machines.

This is much more in-line with how I saw Canadian behave, but it still throws up an issue about server side reporting. If cookies are deleted, the same user can be counted multiple times. With impersonal cookies, it’s also impossible to determine multiple people at a single machine. The next natural argument from the tech side would be to track IP addresses. Unfortunately, with very few personal computers using a static IPs, IP addresses rotate regularly which cause the numbers to be quite inflated. So, server side tracking is not perfect either.

comScore’s solution = hybrid panel + server side reporting.

By using a beacon (a tracking pixel embedded server side that comes from comScore), they will be pulling data directly from participating sites. This data is collected and then run through a filtering process to remove non-human/bot users, to apply census information, to break out reported universes (such as mobile, work, home, etc…) and finally is leveraged against their panelist to provide demographic behaviour data. PHEW!

They have been using this method to test and track sites over the past few months in preparation for a roll out in Canada and the US in August (July 2009 data). One of the case studies that measured a site they believed to be under-delivering (especially amongst at-work audiences) showed a 1.7 x increase with unique visitors and a 2.1 x increase with page views. On a site where they believed tracking to be fairly accurately, their hybrid results showed very little change between the panel reporting and their new hybrid data. Very cool stuff. But what does this mean to the (western Canada) industry?

1) Sudden jump in traffic: comScore is going to try their hardest over the next few months to educate as many people as possible in regards to the recent changes in reporting. They do not want the user community to think that there has been a major change in Internet usage, but rather a correction to reflect correct data. This could potentially cause some headaches and backlash – so do your part. Tell someone who cares today!

2) More granular categorization of publisher content, enabling profiling of these channels against a wide area of planning criteria. This one got me REALLY excited. How would this effect the Prairies? With the additional weight and stabilization of the data, would we be seeing the separation of the 3 amalgamated provinces? I knew that my clients (and partner agencies) in Calgary and Regina continue to be quite frustrated by the combination and it would be huge news to find that this new methodology would allow prairie businesses to take web as a media more seriously. The excellent Steve Rosenblum (IABC’s director of research) was able to put me in touch with comScore Canada president Brent Bernie who replied right away:

“The launch of Panel Centric Hybrid measurement in Canada will certainly benefit smaller sites that emanate from the Prairies if they beacon. As for splitting the provinces this initiative plus our continued panel expansion may allow this level of reporting. It has not been specifically planned so I will take it forward as a client suggestion and see what is possible.”

Yea!! This is such excellent news. I am so glad that I was able to participate and bring this up. It follows that all small-medium websites that feel like they are not getting measured or reported should check out This site *will* contain the beacons that can be embedded to help comScore track local and niche sites. It’s not 100% up yet as they are still working with pre-existing beacon sites to be ready for the launch of the new July data (available August). If you’re hot to get on this going right away (or want a client’s website measured asap), either give me a shout or contact comScore directly.

User note: if you don’t have the time to schedule some training with comScore to understand the impact of the new data, be aware that sites that have an H or a PH beside the data are Beaconed or Partially Beaconed sites. You may see a lift in their numbers or have more granular data available than before. Partial Beaconed sites are those that have only tagged some of their content channels and not their entire site. While comScore is working to get all participating sites 100% beaconed (including video), PH sites will require that the entire content channel be tagged prior to releasing their data. For example, if Yahoo wanted to participate but only wanted to test one channel (say, Sports), the entire Yahoo sports channel would be measured and labeled with a PH. Yahoo Portal data would still be panel based.

For more information about Media Metrix 360, check out the press release here.

  1. First of all, what were the odds that you and I would both end up in industries where we cared about ComScore? Weird.

    Second of all, I second this. ComScore has been extremely problematic, especially, as you mention, for sites that are viewed at work. Hopefully this will help.

    Another interesting dimension is the Google Analytics factor. People are starting to wake up to the fact that GA is 1) Free, 2) A really really good product, 3) shareable with other accounts so that it can be verified. I don’t see GA replacing third party measurement in the near future, but it could give it a run for its money in the long term as it develops. (Heck, maybe Google will eventually add panel measurement to it’s tag-based measurement, coming at ComScore from the other direction!)

  2. Well, I always knew that you would end up in a job in media with your camera-skills and finger on the pulse. I think my studies in Library Science just helped push me towards a job where I could use my nerdy powers of research for good.

    I agree that GA is a huge factor. As usual, Google takes major strides to provide all websites (small or large: one of my big complaints about the old panel-based ComScore reporting) with complete and relevant data. As Google continues to improve the tracking technologies available to the everyone, non-commercial sites will benefit. As comScore continues to improve their tracking technologies, commercial niche sites will finally have a voice. I see a place for both of them – especially if one causes the other to continually better themselves.

    **On a personal note: I would like to applaud and bring your attention to comScore’s participation in Trees for the Future. I am very glad that social responsibility is making it onto the agenda of large (global) corporations.

  3. great website Kathryn. I read the entire Comscore article.

  1. No trackbacks yet.