Inexplicable Shift in Traffic? Consider “Dark Traffic”

Retail and lead generation clients are testing more and more complex attribution models; last click doesn’t give proper credit to the earlier parts of an increasingly-complex customer journey.   While there are a number of ways to calculate attribution, one thing in web analytics has always been a constant: you can count on your ability to differentiate sessions by channel like organic vs. paid vs. direct with a fair amount of accuracy.   Unfortunately, it appears the very thing that is making attribution so difficult – a multi-device, multi-touchpoint sale process – is also making the fundamental association of traffic to channel less accurate.  As a result, any discussion of advanced attribution, including simple last-click channel  ROI, must include a discussion of “dark traffic” .

Dark Traffic Blog Post Image

The Dark Side of Analytics – Dark Traffic

By checking the http referrer of incoming traffic, a website’s analytics application can tell where the request originated and designate that traffic’s source as organic, paid, email, etc.  As Wikipedia explains,

In the most common situation this means that when a user clicks a hyperlink in a web browser, the browser sends a request to the server holding the destination webpage. The request includes the referrer field, which indicates the last page the user was on (the one where they clicked the link).”

If the traffic has a search engine referrer, it is deemed organic. If it has an ad system referrer, it is paid. If there is no referrer information, it is assumed someone typed in your web URL or used a bookmark to reach the website. This method of differentiating channels worked pretty well in the past but, today, an increasing percentage of non-direct traffic does not send http referrer information.  Using the analytics channel attribution methodology we’ve described, you can see that if there is no http referrer, that traffic is put in the “direct” bucket when it may have, in fact, come from another source.

What sources commonly strip referrer information?

Why has dark traffic become a hot topic lately?  Well, because an increasing portion of sites’ traffic has no http header information.  Why?   For a number of reasons:

  1. Visits from secure https-enabled sites to non-secure (like HTTP) sites seldom pass http headers
  2. Visits from social media apps, SMS/chat and other mobile applications seldom pass http headers
  3. Visits from many secure search engines, including image search seldom pass http headers
  4. Visits from links within email, served by Gmail, Yahoo and Outlook seldom pass http headers
  5. Non-web based files, documents, PDFs or presentations seldom pass http headers

Here’s a simple example of how #1, above, could impact your traffic:  let’s say your #1 referring site is http://www.xyzdeals.com and they changed to sitewide https.  Where that traffic formerly appeared in your referral channel within analytics, unless xyzdeals.com does some custom coding, the traffic from their secure site to your non-secure site will now be seen as direct, since secure sites do not send http referrer information by default.

Has your analytics data been impacted?

As the online sales process get more complex, with traffic flowing across different devices, between devices and browsers, apps and websites, and between non-secure and secure sites, dark traffic has risen.   By some accounts, this may impact as much as 60% of your incoming traffic. Other, less extreme examples calculate that dark traffic composes 10-15% of total site traffic.  Generally speaking, the more mobile, social and referral traffic your site receives, the higher percentage of your direct traffic is likely to be “dark”.

Another way to look at it: if you have seen a rise in direct traffic that does not correspond to a promotion and that rise is accompanied by a decline (particularly the same day) in another channel’s traffic, it’s highly likely that traffic shifted to “dark”.

Measuring your site’s dark traffic percentage

The best way to identify the dark traffic coming to your site:

  1. Query analytics for direct traffic
  2. Filter out any direct traffic to your homepage
  3. Filter out return visitors (include new users only) to remove the possibility of bookmarked visits
  4. Consider filtering out any direct traffic to your top category pages

This direct traffic by new users to deep pages is very likely to be “dark”.   Traffic by new users to deep pages is highly unlikely to be typed in (because the URLs are too long) and is highly unlikely to be from bookmarks (because the visitors are not returning). That means it is highly unlikely the traffic is truly direct.

Can dark traffic be prevented?

 While there’s no way (yet) for an analytics platform to automatically attribute dark traffic to its rightful channel, there are ways site owners can correctly identify referrers within the direct channel.   How? Simple: when URLs are distributed to third parties – via email, social, influencer marketing, etc. – make sure it’s tagged to a string parameter associated with the channel. For instance, all links in your email campaigns can be tagged with “?ref=email” at the end of the URL.  (If you’re unsure of proper tagging, refer to Google’s UTM Parameters.)    With these tags in place, you then have the ability to set up a custom report for direct traffic where the entry or landing page has the string parameter you’ve designated.

So there you have it, a quick definition of dark traffic, its common causes and a method for seeing through the darkness.  The real takeaway is that, before you panic about a sudden drop in organic or referral traffic, take a look at direct and see if, perhaps, it has been the recipient of that channel’s proper credit.  Your organic and referral ROI calculations will thank you!

Recommended Posts

Leave a Comment