Search Volume Designed for SEO: Get Over Google’s Bias With Clickstream

Search Volume Designed for SEO: Get Over Google’s Bias With Clickstream

Having accurate search volume data is crucial no matter what kind of business you are in. Why would anyone try selling something if no one is searching for it? The supply-demand nexus is at the very core of the world as we know, and search volume, in turn, is the way to measure customer demand on the web. Whether you are setting up a PPC campaign, doing SEO analysis, creating content, or adjusting product offerings – estimating the traffic for a keyword remains the fundamental element of your workflow this way or another.

The value of search volume data increases dramatically if you look at it from the point of SEO software development. It’s next to impossible to provide any online visibility estimation without accessing traffic that comes with target keywords. So, in other words, having accurate search volume data is vital for any SEO tool, no matter internal or commercial.

What exactly is search volume?

Search volume for a keyword reflects the averaged number of times that the keyword was searched in a month over the range of the past 12 months. A higher search volume shows a higher level of user interest in a keyword, which can represent a topic, product, service, etc. At least, that’s how it should work in a perfect world.

The reality on the ground, however, is different. The specific metrics involved in accessing search volume differ from one source to another and generally represent rough estimates and projections. The calculation is compounded by significant month-over-month and sometimes even day-over-day fluctuations which reflect the volatility of people’s interest in certain topics, products, or services.

SEO’s and digital marketing professionals have historically been gravitating towards search volume provided by Google Ads’ Keyword Planner. That’s only natural, given the fact that Google has been the biggest search engine for decades. Why would you need another source of data if Google has it all?

Well, Google may have it hands down, but the search engine is reluctant to the idea of sharing it with you.

What’s wrong with Google Keyword Planner?

Keyword Planner is a free tool that helps Google Ads users find better keyword ideas for running ads by providing different PPC metrics (including search volume), alongside the suggested keywords. It also gives historical search volume data broken down into months.

However, the data from Google Keyword Planner seems to be getting less and less reliable with each new update, and some of the latest failures have put its value to SEO professionals, and more so SEO software developers, at risk. Let’s review some of the most significant disruptions Google made to the search volume data in Keyword Planner.

1. Broadly matched keywords

About a year ago, Google started aggressively grouping keywords in Keyword Planner, which resulted in unusually high search volume values for lots of keywords. One the one hand, this falls into the logic of determining search intent, which Google has been following for quite some time now. On the other hand, by doing so they disrupted not only SEO but also PPC campaigns by showing misleading search volume and suggested bid data.

For example, if you search for “rent a car uk” in Google Keyword Planner, it would show you information for the “car rental uk” keyword, which has a much larger amount of historical monthly searches.

You can read more about this issue in one of our older articles.

2. Data restrictions

If providing inaccurate search volume weren’t enough, Google also obfuscated this data for users who are running low on Adwords budgets by delivering it in broad ranges. That move made Google Ads platform literally useless for SEOs who were using it only for keyword research and didn’t need to run paid ads at all.

3. Incomplete search volume trends

If you checked historical metrics data in Keyword Planner, you would’ve noticed some inconsistencies in search volume trends it provides, notably the missing data for April. The issue is that Google typically releases search volume for a previous month no more than ten days into the following month. However, it’s the end of May already, but Google hasn’t yet provided data for April.

Important Update: As of June 3, Google finally released search volume data for April.

That, however, may have something to do with recent issues with the Adwords reporting interfaces. But anyway, such inconsistencies only confirm the unreliability of Google Ads in general and its Keyword Planner in particular.

4. Bias against certain keywords

Google Keyword Planner doesn’t provide any metrics for particular keyword categories, such as gambling, healthcare, etc. These keywords represent only a little fraction of searches but can mean a lot for some businesses. Let’s take “paintball guns” for example. Even though the keyword is quite innocent in itself and clearly doesn’t indicate any criminal intent, you won’t find search volume data or any other metrics for it through Keyword Planner.

5. There are only around 80 possible search volume values

If you do large-scale research on search volume values in Google Ads, you’ll discover that figures are often reiterated. Having an extensive keyword database with search volume extracted from Adwords API (our database consists of almost one billion keywords), we were able to determine around 80 search volume values that are repeated across all keywords.

That confirmed our suspicions of Google not being transparent with its Keyword Planner and became the last straw that prompted us to turn to an alternative source of search volume data.

Clickstream Search Volume

Google handles at least 63,000 searches per second and over 167 billion a month, according to Seach Engine Land. All these searches combined underpin Google’s services and, not the least, the calculation of search volume and other keyword metrics.

Now, imagine that you could get your hands on at least a fraction of these 167 billion queries. This would theoretically allow you to get search volume data without relying on Google Ads as a middleman and, who knows, maybe even read people’s minds.

It’s only common sense that Google is the only one who can provide keyword data – either through the Google Ads platform and Adwords API or through companies that scrape data and then resell it. However, that’s only part of the truth.

There are hundreds of applications and browser plugins that are available almost for free. Why almost? Well, because in return for their services, they would ask to collect data on your online activity and then use it for commercial purposes, e.g., resell it. Then, companies like DataForSEO aggregate and structure anonymized data from different clickstream providers and then deliver it to SEO software developers through APIs. Some of the bigger companies that rely on clickstream data are Moz and SEMrush.

Nevertheless, despite all the opportunities it can provide, clickstream data also has its share of flaws and drawbacks. First of all, although its database is massive – a 100-million panel of consumers – it’s still far less than that of Google or other major search engines. Second of all, clickstream databases are messy, so it’s challenging to extract anything useful from them unless you have a team of developers and plenty of time.

The accuracy of the “raw” search volume data obtained from clickstream can also be questionable. Getting the number of searches for a particular keyword isn’t enough – you would’ve to somehow project that data onto the overall number of Google users in a specific location of interest, consider the time when a search session took place, and bear in mind many other variables.

Introducing Bulk Clickstream Search Volume

Here, at DataForSEO, we’ve done a big part of that work for you. Our new Bulk Clickstream Search Volume endpoint of the Keyword Data API can supply you with search volume for up to 700 keywords in a bulk.

You can try it right now by sending a request to the Keyword Data API. You don’t need to submit a separate GET request because all data is provided in real-time. See our docs for more information about formatting the body of your request.

Here’s an example of search volume data that you can get for “buy bitcoin,” “paintball gun,” “buy steroids” and “search volume” keywords.


Request Sample
https://api.dataforseo.com/v2/kwrd_sv_batch_clickstream

 {
   "data":
    [
        {
            "search_engine": "google",
            "loc_name_canonical":"United States",
            "keys":["buy bitcoin", "paintball gun", "buy steroids", "search volume"]
        }
    ]
}


Response Sample

 
{
    "status": "ok",
    "task_id": 281042697,
    "results_time": "0.5834 sec.",
    "results_count": 4,
    "results": [
        {
            "search_engine": "google",
            "loc_id": 2840,
            "key": "buy bitcoin",
            "sv": 79630,
            "ms": [
                {
                    "year": 2019,
                    "month": 4,
                    "count": 36260
                },
                {
                    "year": 2019,
                    "month": 3,
                    "count": 59620
                },
                {
                    "year": 2019,
                    "month": 2,
                    "count": 24220
                },
                {
                    "year": 2019,
                    "month": 1,
                    "count": 38340
                },
                {
                    "year": 2018,
                    "month": 12,
                    "count": 89410
                },
                {
                    "year": 2018,
                    "month": 11,
                    "count": 76490
                },
                {
                    "year": 2018,
                    "month": 10,
                    "count": 67110
                },
                {
                    "year": 2018,
                    "month": 9,
                    "count": 78810
                },
                {
                    "year": 2018,
                    "month": 8,
                    "count": 127550
                },
                {
                    "year": 2018,
                    "month": 7,
                    "count": 121050
                },
                {
                    "year": 2018,
                    "month": 6,
                    "count": 114060
                },
                {
                    "year": 2018,
                    "month": 5,
                    "count": 122620
                }
            ]
        },
        {
            "search_engine": "google",
            "loc_id": 2840,
            "key": "paintball gun",
            "sv": 6560,
            "ms": [
                {
                    "year": 2019,
                    "month": 4,
                    "count": 15200
                },
                {
                    "year": 2019,
                    "month": 3,
                    "count": 11970
                },
                {
                    "year": 2019,
                    "month": 2,
                    "count": 3270
                },
                {
                    "year": 2019,
                    "month": 1,
                    "count": 2890
                },
                {
                    "year": 2018,
                    "month": 12,
                    "count": 4830
                },
                {
                    "year": 2018,
                    "month": 11,
                    "count": 1770
                },
                {
                    "year": 2018,
                    "month": 10,
                    "count": 1760
                },
                {
                    "year": 2018,
                    "month": 9,
                    "count": 14560
                },
                {
                    "year": 2018,
                    "month": 8,
                    "count": 11070
                },
                {
                    "year": 2018,
                    "month": 7,
                    "count": 1730
                },
                {
                    "year": 2018,
                    "month": 6,
                    "count": 6880
                },
                {
                    "year": 2018,
                    "month": 5,
                    "count": 2810
                }
            ]
        },
        {
            "search_engine": "google",
            "loc_id": 2840,
            "key": "buy steroids",
            "sv": 580,
            "ms": [
                {
                    "year": 2019,
                    "month": 4,
                    "count": 0
                },
                {
                    "year": 2019,
                    "month": 3,
                    "count": 290
                },
                {
                    "year": 2019,
                    "month": 2,
                    "count": 0
                },
                {
                    "year": 2019,
                    "month": 1,
                    "count": 730
                },
                {
                    "year": 2018,
                    "month": 12,
                    "count": 0
                },
                {
                    "year": 2018,
                    "month": 11,
                    "count": 0
                },
                {
                    "year": 2018,
                    "month": 10,
                    "count": 0
                },
                {
                    "year": 2018,
                    "month": 9,
                    "count": 990
                },
                {
                    "year": 2018,
                    "month": 8,
                    "count": 0
                },
                {
                    "year": 2018,
                    "month": 7,
                    "count": 330
                },
                {
                    "year": 2018,
                    "month": 6,
                    "count": 860
                },
                {
                    "year": 2018,
                    "month": 5,
                    "count": 250
                }
            ]
        },
        {
            "search_engine": "google",
            "loc_id": 2840,
            "key": "search volume",
            "sv": 830,
            "ms": [
                {
                    "year": 2019,
                    "month": 4,
                    "count": 1510
                },
                {
                    "year": 2019,
                    "month": 3,
                    "count": 810
                },
                {
                    "year": 2019,
                    "month": 2,
                    "count": 290
                },
                {
                    "year": 2019,
                    "month": 1,
                    "count": 260
                },
                {
                    "year": 2018,
                    "month": 12,
                    "count": 0
                },
                {
                    "year": 2018,
                    "month": 11,
                    "count": 340
                },
                {
                    "year": 2018,
                    "month": 10,
                    "count": 0
                },
                {
                    "year": 2018,
                    "month": 9,
                    "count": 2800
                },
                {
                    "year": 2018,
                    "month": 8,
                    "count": 0
                },
                {
                    "year": 2018,
                    "month": 7,
                    "count": 330
                },
                {
                    "year": 2018,
                    "month": 6,
                    "count": 300
                },
                {
                    "year": 2018,
                    "month": 5,
                    "count": 0
                }
            ]
        }
    ]
}

 

Now, let’s take a closer look at the figures we obtained and compare them with Google Keyword Planner’s.

  • Keyword
      • Buy bitcoin
      • Paintball gun
      • Buy steroids
      • Search volume
  • Clickstream
      • 79,630
      • 6,560
      • 580
      • 830
  • Keyword Planner
      • 60,500
      • 0
      • 0
      • 590

As you could’ve guessed, Keyword Planner didn’t provide any results for “paintball gun” and “buy steroids” keywords. As for the remaining two keywords, the deviation of the values doesn’t exceed 30%. Given that Google rounds down the search volume data it shows, this discrepancy is rather negligible. What’s more, clickstream may be even more accurate than the Keyword Planner.

Here’s how Vladislav Omelchenko, the Principal Software Engineer at DataForSEO, explains it:

“The main issue with search volume that Google {Keyword Planner} provides, is that it’s calculated proceeding not only from a certain search query, but also from different groups, categories, and other unknown multipliers. But we needed to access the “root” value of a keyword and that’s how we turned to clickstream data. By using the statistical analysis and applying click data for a massive number of users, we designed a model for determining a real search volume for a particular keyword. The figures we provide in Bulk Clickstream Search Volume API reflects the approximate averaged number of unique search sessions that involved a keyword in the selected search engine and location.”

Search volume designed for SEO

Ultimately, we can’t say the DataForSEO’s clickstream-based search volume data is a substitute for what Google Keyword Planner provides – don’t expect it to give you the same numbers as Google. However, given all the Keyword Planner issues that have only been snowballing throughout the years, clickstream data is the only sustainable alternative that comes to mind.

DataForSEO’s Clickstream Search Volume API is the most accurate commercially available solution for getting search volume data outside of Mountain View. But don’t take our word for it – you can try it out for free!

TRY IT FOR FREE

No Comments

Post a Comment