October 18, 2010 » 5 Web Analytics Misconceptions: Are You a Victim?

The Internet is overflowing with metrics that are collected and analyzed without a thought as to how they are calculated and the limits of what they infer. Web Analytics tends to use misleading language, e.g., “time” and “unique” metrics; pair this with a misunderstanding of the collection of these metrics and it leads to mistakes in data collection, misunderstanding of specific metrics, and incorrect analysis.

Data Collection:

1 – Tracking Tags cannot Time Travel

Tracking in place at launch: Excellent!

Tracking tags are the snippets of code that need to be added to websites, varied functionalities within websites, and other online initiatives that allow a Web Analytics or Ad Server tool to collect and analyze data. Occasionally site functionalities launch without the tracking necessary to best capture the desired behavior. Inevitably this leads to the question: “If we implement the tags now can we report on the data since launch?” The most popular offenders in this case are usually data points beyond standard page tags, e.g., video or flash content.

It is possible to draw conclusions once tags are in place and the behaviors surrounding the functionality are tracked, it is not optimal. The data that was not tracked can never be recovered. This highlights the importance of the implementation and QA of tags prior to launch. Tracking tags should be treated as if they were critical to the success of your project (because they are).

Specific Metrics:

2 – Time on Site/ Page is not a Stopwatch

How a metric is calculated is important

Time on Site/ Page is a popular metric to measure engagement. It is usually thought to be collected as if by a stopwatch that starts the moment a user sees the first page on the site and stops the moment that user closes the browser or navigates away from the site.

Time on Page is calculated by subtracting the time from the server request for “Page A” from the server request for “Page B”; the difference is the time spent on “Page A”. This cannot be calculated for the last page of the site visit because there is not a second server request. Time on Site is a sum of all the Time on Pages for that Visit.

Imagine a user looks at the first page of their site visit (“Page A”) at 12:00 noon, accessed their second page (“Page B”) at 12:02 PM, spends 20 minutes on “Page B”, and then exits the site. Time on Page for “Page A” is 2 minutes, for “Page B” is 0 seconds, and Time on Site is 2 minutes. Ironically, the most valuable time on the site is missing from the data. Due to this inaccuracy, Time on Site/ Page should be taken with a grain of salt. Time metrics are much better suited to analyze video content (calculate the percent complete of the video and the attention span of your users).

3 – Unique Visitors does not equal Unique People

Just because you LOOK like one person doesn’t mean you are

Unique Visitors is largely understood to stand for unique people; this is not accurate. When a user accesses the site their browser receives a cookie. This cookie is used to track behaviors on the site. Unique Visitors, in most analytics tools, is a count of those cookie-browser combinations.

This can be inaccurate in three scenarios:

1. One person using multiple browsers/ computers; for example, one person uses a work and a home computer to access the site or one person uses Chrome and Firefox on one computer to access the site. These make one person look like two Unique Visitors.
2. Multiple people using one browser; for example, a patient and a caregiver accessing the site from the same browser on the same computer. This makes two people look like one Unique Visitor.
3. When people delete their cookies between visits to the site another cookie is assigned to them. This makes one person look like two Unique Visitors.

The only way to track Unique Visitors more effectively is with a registration-only site section. Then multiple cookies can be linked together with a common username/password. Due to the above scenarios Total Site Visits should be utilized as a measure of volume.

Analysis:

4 – Quantity of Page Visits does not equal the Quality of the Content

A common metric to judge site content is Page Visits (page views), which is largely understood to show pages that provide the most value to users. However, the quantity of Page Visits does not equal the quality of the content.

More is not always good

The quantity of Page Visits is affected by different variables, including:

1. Landing page popularity due to paid search and other media. High quantity here means paid media success, not quality content.
2. The page is a “Router” page (first page in a site section). High quantity here means site section popularity, not quality content on the Router page.
3. Best of the rest scenarios where users cannot find what they were originally looking for. For example, giving users an option between drug content or disease content; then assuming that whichever they choose is what information is most important. At most, it is clear that they favor one over the other. High quantity here means best content available (or the most visible), not the highest quality content the user would have liked to see.

It is possible, but not necessary, for pages with a high quantity of Page Visits to contain high quality content. Pages that are of high quality may be hidden on the site and not accessed in large numbers. Also, note that confusing quantity with quality is often made with other metrics as well, e.g., Total Site Visits, Page Views/ Visit, and Time on Site/ Page.

5 – Traditional Web Analytics cannot provide Insight into WHY

Traditional Web Analytics is great at answering “What”, “Where”, “When”, and some of “Who”; however, it cannot provide insight into “Why”. We can tell what users do on the site, where they were located, when they visited the site, and some of who people are – based on content viewed and any registration processes.

Always hunt for WHY

We tend to fill in the gaps of data to find the “Why” that makes the most sense or tells the best story. Without additional data we may fall victim to confirmation bias: a tendency to favor information that confirms preconceptions or hypotheses. If users are bouncing from a specific page in high percentages and we try to answer “Why”, the response may be affected by confirmation bias.

Ask someone who built the site and bounces are not necessarily a bad thing: “Users got all the information they needed on that one page and then left.” Ask that same person to analyze a potential client’s site and bounces are bad: “Users did not like the page because they exit right away.” Another common question is: “Why are people coming to the site?” Confirmation bias would allow people to fill in that gap with any number of responses.

Conducting user surveys on the site fills in gaps left by “Why” questions and avoids confirmation bias. The best way to answer: “Why are people coming to the site?” is to ask people. Connect those answers with positive site satisfaction and you find the high quality content. Instead of wondering if bounces are bad, ask the users why they bounced.

Now it’s Your Turn

Have you been a victim to any of these misconceptions? Were you aware of these already? Are there other Web Analytics misconceptions that you have had to learn the hard way?