Backlink APIs: An In-Depth Comparison of DataForSEO, Ahrefs, and Semrush

The landscape of backlink data providers is continually evolving, with new players, tools, and methodologies emerging in the arena. This dynamic market environment fuels the ongoing quest for the most comprehensive, accurate, and high-quality solution.

To assess our position in this fast-paced landscape, DataForSEO decided to undertake an in-depth comparison of our Backlinks API alongside other prominent backlink data vendors: Ahrefs, and Semrush.

In this article, we’re inviting you to dive into the results and related insights.

Contents:

How to Measure Backlink Index Quality: Beyond the Size
Comparing DataForSEO, Ahrefs, and Semrush Backlink Indexes
Index Size, Freshness, and Historical Data Retention
Comparison of Reported Backlink Stats
Link Diversity
Spam Detection and Proprietary Authority Metrics
Cost Efficiency and Additional Factors
Summary

How to Measure Backlink Index Quality: Beyond the Size

From the customer’s perspective, it is essential to evaluate several backlink data providers side by side and make sure the data you rely on is trustworthy.

Yet, how to recognize a meaningful comparison of backlink indexes, and what points to consider?

Common approaches (the bad practices)

If you have scrolled through one or two similar articles, you must have noticed they tend to focus on the size of backlink indexes. There are several reasons behind this unfortunate tendency:

➤ It’s easier. The size of a backlink index is a simple metric that potential customers can quickly understand. At the same time, analyzing the index’s freshness, spam detection features, and other complicated factors is far more nuanced and requires a deeper understanding of the data.

➤ It might help marketing. Given that index size easily communicates the database volume to potential customers, it is often used as an attention-grabbing marketing tool and demonstrated as a measurable competitive advantage.

➤ It could cater to user expectations. The backlink index size does allow users to understand the provider’s potential to cover a broader online landscape. Some users may assume that a larger index will be more advantageous for their research or development purposes, so data providers try to meet this demand by focusing on the index size in their marketing materials.

➤ There’s limited information available. In a competitive market, providers cannot disclose certain information about their indexes and data collection process, so it is impossible to assess some qualitative aspects.

Nevertheless, focusing solely on the index size is a misleading practice as it does not offer a comprehensive picture of the contained data’s quality.

Comprehensive approaches (the good practices)

While index size may be a useful starting point, it is essential to consider factors that actually help to measure the quality and credibility of the offered data. Here are the key evaluation criteria which may vary based on your specific requirements:

➤ Freshness. A good index is regularly updated to include new backlinks and remove non-existent ones to serve you with the most current information.

➤ Historical data. Long-term data retention and the opportunity to analyze changes over time indicate a quality backlink index.

➤ Link diversity. Different types of backlinks, including dofollow, nofollow, redirect and others should all be appropriately marked in a comprehensive index.

➤ Data granularity. The index should deliver granular backlink data insights that align with your specific requirements.

➤ Spam detection. The provider should offer an option to filter out spammy or low-quality links.

➤ Proprietary authority metrics. Provider’s proprietary metrics can offer a more nuanced assessment of a domain’s or page’s trustworthiness instead of basic statistics like the number of backlinks, making it easier for users to navigate the datasets.

➤ Cost efficiency. The pricing of a backlink index should correspond to the value it provides.

To compare backlink data providers in terms of value for money, refer to our earlier blog post >>

We hope that by this point, you understand why it is important to look beyond common approaches to backlink index comparison, such as overreliance on the index size. Instead, it is worth going the extra mile and exploring more diverse factors that offer a holistic view of the backlink index value.

Comparing DataForSEO, Ahrefs, and Semrush Backlink APIs

Based on the insights from the previous section, we’ve established a set of general (not case-specific) criteria for our backlink index comparison of three prominent market players: DataForSEO, Ahrefs, and Semrush. In the parts that follow, we will take a closer look at each provider’s strengths and weaknesses in terms of:

➤ Index Size, Freshness, and Historical Data Retention
➤ Reported Backlink Stats
➤ Link Diversity
➤ Spam Detection and Proprietary Authority Metrics
➤ Cost Efficiency and Additional Factors

Our goal is to dissect all nuances and provide you with an objective comparison to help you make a well-informed choice of a backlink API provider for your specific needs.

Backlink Index Size, Freshness, and Historical Data Retention

As we’ve already mentioned, different providers take different approaches to calculate their index size.

For instance, DataForSEO bases its 2.8T Backlink Index only on live backlinks currently present in the database. We store both live and lost backlinks, with lost backlinks remaining accessible for up to one year after they were marked as lost. If we calculate the total number here, it makes for around 4T backlinks. Importantly, we understand backlinks as links from one domain to another, also known as “external backlinks” or “inbound links.”

Our database also contains 336.3B indexed pages and 289.5M domains, but it only includes domains that have at least one backlink and does not include backlinks that were removed more than one year ago.

It’s worth noting, that our index architecture is designed to provide access to historical data separately from fresh stats. You can obtain historical backlink data back to the beginning of 2019 – that’s when we started developing our index. Lastly, we do not include historical statistics in the calculation of our backlink index volume, but if we did, this would make for over 6T backlinks in total.
‌
Ahrefs, on the other hand, states their index stores historical data for 35T external backlinks and contains 345.2B pages and 210M domains. However, their Historical Index is “a record of all backlinks that were seen by Ahrefs crawlers as “live”, but not necessary live today” and they started recording data in this index in May 2016. Ahrefs also has a Recent Index consisting of “all “live” backlinks plus “lost” backlinks that had “live” status during the last 3-4 months” and a Live Index that “contains “live” backlinks only” but these indexes’ volumes are undefined.

Semrush, with a reported 43T backlinks, does not provide historical data but offers a “Fresh Index” based on the backlinks that its bot observed in the last 6 months of crawling the Internet. They also indicate 1.6B referring domains contained in their index. However, this is a significant disparity compared to an estimated total of 354M domain name registrations in the first quarter of 2023 across all top-level domains as recently reported by Verisign. This suggests that the numbers reported by Semrush should be considered with caution when using such data for analysis or decision-making.

Given all of the above, it does not make much sense to compare the reported index sizes unless we take into account the definitions and criteria used by the providers to produce this metric. It might also be more useful to evaluate the quantitative characteristics of backlink indexes collectively, including volume, update frequency, and historical data retention period.

Here’s a table detailing these characteristics as stated on the providers’ websites.

Additionally, to understand the indexes’ growth and refresh rate potential, we can suggest that you consider impartial statistics from Cloudflare surfacing the ranking of the most active bots from the SEO category.

The Rank column here indicates the cross-category rank among all verified web bots. For example, as of October 24, DataForSEO is listed the third in the SEO category and is ranked the 22nd most active good bot across all categories out of 172 bots. Our goal is to top the SEO industry rating, and while we are on that journey, we should say we’re pleased with the progress we’ve made so far, as we launched the DataForSEO Link Bot only four years ago.

Lastly, we wanted to mention that the review of the reported index volumes did not allow us to objectively compare the coverage of the analyzed providers, so we decided to conduct our own study using comparable data.

Reported Stats from DataForSEO, Ahrefs, and Semrush Backlink Indexes

When setting out to compare the backlink statistics reported by DataForSEO, Ahrefs, and Semrush, our primary objective was to understand to what extent the providers’ data aligns while ensuring data consistency. Unlike typical index studies, our comparison approach is centered on minimizing the disparities between the indexes. We highly recommend that you carefully consider our methodology before interpreting the results.

Methodology

Analysis criteria: we concentrated on live (active) backlink data, representing the current state of backlink statistics, and the root domain mode, examining data for target domains including their subdomains.

Date of the analysis: January 22, 2024.

Comparison metric: the number of live main (root) referring domains.

We have chosen this metric as it allows for a more transparent view of the data in contrast with comparing the reported number of backlinks, which could lead to miscalculations.

It’s important to note that different providers have distinct approaches to counting backlinks.

For example, DataForSEO’s Backlinks endpoint provides and counts unique backlinks from referring pages. If a page contains more than 1 backlink pointing to the target, the link with the highest rank is returned and counted.

At the same time, Ahrefs returns multiple backlinks from the same page and counts them separately, providing an option to group similar links.

We’ve seen the same situation in Semrush, however, Semrush does not offer a filter to group similar links, allowing only to group the results by one, two, or ten backlinks per referring domain.

Having said that, all providers indicate they consider a domain pointing at least one live backlink to the target domain as a live referring domain. Importantly, we’re using a common metric across DataForSEO, Ahrefs, and Semrush – main (root) referring domains. It does not count referring subdomains as separate referring domains.

Dataset: For this analysis, we picked the last 10 domains from the Most Visited Environmental Science Websites Worldwide. As of the date of the analysis, this list was last updated on January 1, 2024, according to Similarweb.

Limitations and associated challenges: We have to acknowledge that several uncontrollable factors may impact this as well as any similar analysis.

➤ Backlink crawling and indexing approaches. Every provider uses distinct methods to index backlink data. For example, DataForSEO does not support the indexing of frame links, while Ahrefs and Semrush seem not to include meta-refresh links. We will review this further in this article. Additionally, DataForSEO employs internal filtering to remove low-quality or irrelevant backlinks and referring domains. We also use a custom minimum quality threshold to avoid indexing URLs and domains that do not pass it, and so does Ahrefs. However, we did not find information stating that Semrush follows such a practice.

➤ Timeframes. Lastly, the providers we analyze have distinct timeframes for which their indexes store live data. It also takes a different timespan for each provider to recrawl their data records and update them. Apart from using the information stated by the providers themselves on their websites (as we mentioned here), it is impossible to objectively estimate all the timeframes related to the data used in this analysis.

Reported values and key findings

To present our findings, we have compiled a comparison table below showcasing live data reported by DataForSEO API, Ahrefs API, and Semrush interface. In technical terms, the given values correspond to:

➤ DataForSEO’s referring_main_domains from the Bulk Referring Domains endpoint which returns live data.

➤ Ahrefs’ live_refdomains returned in the Backlinks stats endpoint.

➤ Semrush’s total count from the Referring Domains report in the interface with an active filter applied. Unfortunately, as of the date of this comparison, there is no way to retrieve data with such a filter using the Semrush API.

Table of referring main domains reported by DataForSEO, Ahrefs, and Semrush for ten target websites from the Environmental Science category

Legend:
. – the largest reported value; . – the second largest reported value.

As we explore the data, it becomes evident that Semrush returns the largest numbers in the majority of cases, while DataForSEO predominantly captures the second-highest reported numbers, closely trailed by Ahrefs. Notably, the disparity in the reported numbers of referring domains initially seems pronounced, but it diminishes considerably relative to the overall decrease in reported figures, as illustrated in the comparison chart below.

Comparison chart of DataForSEO, Ahrefs, and Semrush by the number of reported live referring domains for ten target websites

In summary, our analysis reveals a closely contested landscape where all providers demonstrate strong coverage on the example of the analyzed websites. It’s worth noting that the impossibility of retrieving live (active) data with the Semrush API presents a considerable limitation for granular backlink analysis. With these insights in mind, we recommend looking beyond the comparison of reported stats to choose a suitable provider.

Yet, before we get there, we invite you to run your own analysis using the steps outlined below.

Note that to avoid data inconsistencies, make sure you use the same or closely matching settings/filters across all providers you wish to compare.

To reproduce this research with DataForSEO’s graphic user interface, register and head to the Backlinks API Explorer section. Then, select Bulk Referring Domains under Functions and specify your target domains separated by a comma in the Targets field below as shown on the screenshot. Click Request. The response will appear in the code box on the right with the number of root referring domains for each target indicated in the referring_main_domains field.

Example targets:
ipcc.ch, carbonbrief.org, climate.gov, miteco.gob.es, ethicalconsumer.org, dnv.com, mem.gov.cn, conacyt.mx, scielo.org.bo, justdiggit.org

To reproduce this research with DataForSEO API, use the cURL below to make an API call to the DataForSEO’s Bulk Referring Domains endpoint. It returns numbers based on all live referring domains, that is, the total number of domains pointing to the target with any type of backlinks (e.g., nofollow, noreferrer, UGC, sponsored, etc) that were found during the latest check. The endpoint returns results for the root domain (domain with all of its subdomains).

Instead of ‘login’ and ‘password’ use your credentials from the DataForSEO Dashboard.

login="login" 
password="password" 
cred="$(printf ${login}:${password} | base64)" 
curl --location --request POST "https://api.dataforseo.com/v3/backlinks/bulk_referring_domains/live" 
--header "Authorization: Basic ${cred}"  
--header "Content-Type: application/json" 
--data-raw "[
    {
        "targets": [
            "scielo.org.bo",
            "miteco.gob.es",
            "ipcc.ch",
            "dnv.com",
            "carbonbrief.org",
            "climate.gov",
            "mem.gov.cn",
            "justdiggit.org",
            "ethicalconsumer.org",
            "conacyt.mx"
        ]
    }
]"

Use the results indicated in the referring_main_domains field of the API response. This parameter shows the number of referring main (root) domains pointing to the respective target.

To reproduce in the Ahrefs interface, make sure you set the mode to subdomains and choose the “Don’t show” setting in the History filter to obtain live data. Repeat for every target.

To reproduce with Ahrefs API, use the following cURL with your API key. Make sure you set mode=subdomains. Repeat the request for each target.

curl --request GET \
--url 'https://api.ahrefs.com/v3/site-explorer/backlinks-stats?target=ipcc.ch&mode=subdomains&date=2023-10-25&output=json' \
--header 'Accept: application/json, application/xml' \
--header 'Authorization: Bearer YOUR_API_KEY'

Use values from the live_refdomains parameter in the API response.

To reproduce with the Semrush interface, make sure you set the Root domain mode and select an “Active” filter. Repeat for every target.

Unfortunately, this comparison cannot be performed using Semrush API due to the absence of the necessary functionality for obtaining data on active referring domains.

Link Diversity in DataForSEO, Ahrefs, and Semrush Backlink APIs

In this section, we will consider how DataForSEO, Ahrefs, and Semrush stack up against each other in terms of link type diversity.

Link type is one of the essential factors for understanding which links pass authority signals to search engine bots and which don’t. While it’s true that all three providers return both dofollow and nofollow backlinks, there’s a broader link variety that may be important for case-specific needs. Let’s take a look at what DataForSEO, Ahrefs, and Semrush Backlink APIs offer based on their documentation.

You can also explore this information in the providers’ documentation by these links:

➤ DataForSEO Backlinks API documentation
➤ Ahrefs Site Explorer API v3 documentation
➤ Semrush Analytics API documentation

In conclusion, Ahrefs covers the most extensive variety of backlink types, making it the clear choice for users with high link detail requirements. DataForSEO also provides a great backlink type diversity and includes a meta refresh type not supported by other providers. At the same time, in this comparison, Semrush offers a more limited selection.

Spam Detection & Authority Metrics in DataForSEO, Ahrefs and Semrush API

In the quest for quality backlink data, measuring only its breadth and depth does not do the trick. What’s truly indispensable is to have the instruments to separate the wheat from the chaff, that is, to tell valuable backlinks from potentially harmful ones. With that in mind, we will now delve into the comparison of spam detection capabilities and backlink trustworthiness metrics offered in DataForSEO, Ahrefs, and Semrush APIs.

DataForSEO

At DataForSEO, we have a proprietary Spam Score metric designed to help our users reduce the amount of manual work. It sheds light on the spam level of a page or backlink, a group of pages, or even an entire domain, using a scale from 0% to 100% segmented as follows:

➤ 0%-30%: Low Spam Score
➤ 31%-60%: Medium Spam Score
➤ 61%-100%: High Spam Score

To calculate this Score, we have an algorithm in place that automatically evaluates each scanned page based on 18 meticulously selected signals that include domain name length, the ratio of external links to internal links, and protocol usage (HTTP or HTTPS). To determine the Spam Score for a group of pages (backlinks_spam_score) or an entire domain (target_spam_score), we first calculate this Score for each individual page and then find the average.

In the DataForSEO Backlinks API, we provide a specific Bulk endpoint that allows you to get Spam Score values for up to 1,000 domains, subdomains, and pages in a single request. Besides that, this metric is returned in the responses of over 8 endpoints, including Summary, History, Backlinks, and Referring Domains.

The DataForSEO’s powerful trio of Rank values for a domain, page, or backlink is also included in the responses of numerous endpoints and can be obtained via a separate Bulk Ranks endpoint for up to 1000 targets. Our calculation formula is based on the method for node ranking in a linked database – the core of Google’s original PageRank algorithm, so the metrics we return can become an acceptable alternative to PageRank. The Rank score ranges from 0 (no backlinks detected) to 1,000 (highest rank), with higher values more challenging to attain.

Ahrefs

Ahrefs does not offer an automatic approach to detect “spammy” backlinks leaving the users to employ the hands-on method. Depending on your needs, manual spam detection could be a good option, allowing for high scrutiny and selection of custom spam markers. However, it may be rather resource- and time-intensive for large projects.

As for the proprietary metrics, Ahrefs has two of them – Ahrefs Rank and Domain or URL Rating. The first one represents an ordered ranking of all websites indexed by Ahrefs according to their “size and quality of their dofollow referring domains — basically, by their DR.”

The DR refers to the second metric – Domain Rating – which shows a website’s strength on a 100-point scale. This value is “dependent on not only the total number of high-DR sites that link to you, but also on how many other websites these high-DR sites link to.” Ahrefs URL Rating (UR) analyzes the same factors at the page level.

In Ahrefs API v3, there’s a separate Domain rating endpoint allowing you to obtain both Ahrefs Rank and Domain Rating for a certain domain, but these values as well as the URL Rating are also available via the Backlinks, Broken Backlinks, and Refdomains endpoints.

Semrush

Semrush serves a Toxicity Score in their interface – a metric covering over 45 different markers of toxic backlinks. However, unfortunately, this Score is not available in the API responses, which means users should apply the same approach as with Ahrefs API to detect potentially spammy backlinks – manual research.

As for the Authority Score, this metric grades the overall quality of a webpage or a domain considering its Link Power (quality and quantity of backlinks), Organic Traffic (estimated monthly traffic), and Spam Factors (manipulation or spam in the link profile). In the API responses, Semrush’s Authority Score is represented numerically with values closer to 100 meaning higher trustworthiness of the domain or page. You can obtain this metric in the majority of API endpoints.

To sum up, DataForSEO, Ahrefs, and Semrush each developed unique approaches to assess the credibility of backlinks, referring domains, and pages. Their ready-made proprietary authority metrics allow users to swiftly differentiate potentially beneficial links and resources from harmful ones.

When it comes to tackling spam detection challenges, DataForSEO with its Spam Score emerges as the top choice for users seeking a fully automated and comprehensive solution for detecting low-quality and spam backlinks. In contrast, Ahrefs, and Semrush do not serve a similar metric in the API responses, offering users to rely on their authority metrics as an alternative or to adopt manual spam detection.

Cost Efficiency of Backlink APIs and Additional Factors to Consider

In the search for your ideal API provider of backlink data, the last but not least important point to consider is the balance between your budget and the quality of data and service you receive.

Let’s start with the cost estimations. As each provider applies a distinct pricing model, it is worth understanding the details.

DataForSEO

DataForSEO offers Backlinks API at an affordable entry point of $100. This is a minimal monthly commitment that is redeemed to your account balance and can be used on any DataForSEO API, including Backlinks API. We do not charge for any specific parameters in the API results or filters used in the body of your request.

Here’s how the cost is formed.

Price per single Backlinks API request: $0.02
Price per row of backlinks and related data: $0.00003
Limit of rows per request: 1,000
The total cost of one task with 1,000 rows in US dollars: 0.02 + 0.00003 * 1000 = $0.05

See DataForSEO pricing >>

Ahrefs

In Ahrefs, the cost of data retrieved with API is expressed in units. Here’s how Ahrefs explains its pricing model:

“The base_cost of each request is 50 units. The per_row_cost is the sum of the costs of each unique field that appears in either the returned result, or in the where or order_by parameters. If a field appears in multiple parameters, it is counted once. Most fields have a default cost of 1 unit. However, certain metrics are more expensive and may consume 5 or 10 units.” Source.

Ahrefs additionally states that if the total row cost (per-row cost multiplied by the number of rows) is less than the base cost, such a request costs the base amount of 50 units. However, if the total row cost exceeds the base cost, then the total row cost is charged.

Given that the Enterprise plan ($999/mo) includes 2M API units per month, we can calculate an approximate cost of 1 unit as follows: $999 / 2,000,000 = $0.0004995.

Now, let’s make a cost estimation, of getting 1,000 backlinks supposing we select 10 basic rows of related unique fields to return.

Per-row cost: 10 fields * 1 unit = 10 units.
The total number of rows (backlinks with related data) to retrieve: 1,000.
Total row cost: 10 units * 1,000 rows = 10,000 units.
The total cost of one task with 1,000 rows in US dollars: 10,000 units * $0.0004995 = $5.00

Semrush

The cost of data at Semrush is formed as follows:

“When running an API request, each particular line or request will cost API unit. The total amount of units that you will need to spend per request will depend on the exact number of lines in the report or request.” Source.

One line of data in Semrush is the same concept as one row of data in DataForSEO and Ahrefs – one backlink with related data. The units in Semrush have to be purchased separately on top of the Business plan subscription.

“One dollar is equal to approximately 20,000 API units.” Source.

Based on this, we can estimate that 1 unit costs: $1 / 20,000 units = $0.00005

Backlinks endpoint pricing: 40 API units per line (40 units * $0.00005 = $0.002)
Lines to retrieve: 1,000
Total units consumption: 1,000 * 40 = 40,000 units
The total cost of one task with 1,000 rows in US dollars: 40,000 units * $0.00005 = $2

However, if you do not need to use the Semrush platform for which you should pay $449.95/mo anyway, don’t forget to add that charge on top when estimating your monthly expenses.

For a quick roundup here, you can find the main information in the table below.

Now, we’ll take a look at the customer support and trial period offered by DataForSEO, Ahrefs, and Semrush.

In these aspects, DataForSEO excels with 24/7 live chat assistance and a free unlimited API trial period. You can test the Backlinks API via API Explorer until your trial funds run out. Trial funds on the account balance ($1) are provided by DataForSEO upon registration. Additionally, you have the option to test the Backlinks API in the application of your choice for 14 days without a subscription. After trial activation, you’ll be charged based on a PAYGO model.

Both Ahrefs and Semrush also offer quite robust customer support, though it is limited to business days (Ahrefs) and business hours (Semrush). As for the free API trial, both providers do not offer such an option for new users.

In Ahrefs, “users on any paid plan can test Ahrefs API by running a limited set of free test queries.” At the same time, it is impossible to get free platform access for a trial.

Semrush has a slightly different approach, providing a free 7-day trial of the platform features, however, limiting the plans available for trial to those with no API access. Accordingly, you can only get a Business plan and purchase API units to test the product.

All of the considerations above are summarized below for convenience.

In conclusion, DataForSEO stands out with a lower starting price and competitive cost per 1,000 backlinks along with a free trial and round-the-clock support. While customer assistance at Ahrefs and Semrush is available within acceptable schedules, these providers do not excel at other aspects. Ahrefs has a high-set pricing and its API trial access is limited to paid users of lower plans. Semrush offers a moderate pricing model, yet does not provide free API testing.

Summary

In our comprehensive comparison, we have thoroughly examined DataForSEO, Ahrefs, and Semrush across a range of dimensions, including index size, historical data retention, reported backlink statistics, spam detection, proprietary authority metrics, cost-effectiveness, and more.

We’ve discovered that:

➤ There’s no point in comparing index sizes without considering the providers’ specific definitions and counting methods for this metric.

➤ In terms of reported backlink stats, Semrush returned the highest numbers, with DataForSEO following closely, and Ahrefs coming in third. However, the impossibility of retrieving live data through Semrush API is a significant obstacle to in-depth analysis.

➤ Ahrefs covers the most extensive diversity of backlink types.

➤ Spam Score in DataForSEO’s Backlinks API stands out as a comprehensive and automated solution for identifying low-quality and spam backlinks, while Ahrefs does not offer a similar metric and Semrush does not return its Toxicity Score in API responses.

➤ All three API providers offer unique and carefully developed metrics for measuring the credibility of backlinks, referring domains, and pages.

➤ DataForSEO excels at offering the most attractive starting price, along with affordable cost per 1,000 backlinks, free trial, and round-the-clock support.

Here’s a more detailed summary of all the points discussed in this article.

On a final note, we hope our article has equipped you with the insights needed to avoid falling for some aggressive marketing tactics or misleading backlink index studies. Feel free to expand our list of backlink API evaluation criteria with more points tailored to your specific needs. Lastly, remember that carefully choosing a backlink data provider is key to securing a quality service that aligns with your requirements and thus making a cost-effective investment.

If you’d like to further explore DataForSEO’s Backlinks API or reproduce our data analysis, you’re welcome to register for a free trial and use our API playground.