Good bot for a fair play
What is DataForSEO Link Bot?
Many people would classify DataForSEO Link Bot as yet another web crawler.
Well, they are wrong, and here’s why.
Ever since we launched DataForSEO, our mission was to level the playing field by providing
quality data to SEO enthusiasts and professionals around the world.
DataForSEO Link Bot is key to ensuring that every member of the SEO community – no matter
the size of the pocket – can access the core element of Google’s ranking algorithm.
Yes, we’re working on the biggest available backlink database on the web that every single
member of the community, including you, can use and benefit from.
DataForSEO Link Bot
Version: 1.0
Bot type: backlink checker bot
Obeys robots.txt: YES
Obeys crawl delay: YES
User agent string:
Mozilla/5.0 (compatible; DataForSeoBot;
+https://dataforseo.com/dataforseo-bot)
Enter the Backlinks API
Backlinks API is the name of our new product that has been developed with a single purpose: providing website owners, webmasters, and SEO professionals with opportunities to analyze the key component of website optimization – backlinks.
To attain this ambitious goal, we had to build a system similar to Google’s PageRank algorithm and make sure it works. Our team had spent the past year researching, planning, building, and iterating this system.
But to go any further, we’ll need your help. No, this time we don’t want to sell you anything. In fact, the best thing you can do is… well, do nothing.
If you’re reading this, don’t rush straight to your robots.txt to block our user agent. This way, you will help thousands of website owners – like yourself – to improve their websites’ visibility.
Modify DataForSEO Bot
for a smoother ride
There are literally thousands of good bots crawling your website to help parse and index your content and make it available to users searching for it.
DataForSEO Bot belongs to the category of Good Bots, meaning it won’t harm your website in any way. Our bot supports your robots.txt settings, explicitely identifies itself and has an official moniker.
You have a full control over what DataForSEO Bot can and cannot do on your website. For example, you can change the default 5-second frequency of it visiting your site or block it from accessing certain pages or folders by using wildcards (*).
Change crawl delay of DataForSEO Bot
User-agent: DataForSeoBot
Crawl-delay: 5
Disallow DataForSEO Bot on a page
User-agent: DataForSeoBot
Disallow: /example-page
Request the force-reload of robots.txt
Want to learn more about DataForSEO Bot?
Contact our support team to request more information.
Quick FAQ
DataForSEO Bot is constantly crawling the web to add new links to our backlink database and check links that are already included.
This is crucial to support our mission – providing website owners, webmasters and SEOs with equal access to vital data.
DataForSEO Bot crawls your website pages and adds all found links to our backlink database. It does not collect any other data about your website and won’t trigger ads or add numbers to your traffic in Google Analytics.
DataForSEO Bot is a good bot that plays by your rules. It allows modifying lots of crawling parameters, including crawl delay. Add the following snippet with a crawl delay you need to your website’s robots.txt file:
User-agent: DataForSeoBot
Crawl-delay: [value]
DataForSEO Bot relies on a number of subnets to collect backlink data. You can use these subnets to identify DataForSEO Bot. The following are IPv4 and IPv6 identifiers of our subnets.
IPv4:
136.243.220.208/29
136.243.228.176/29
136.243.228.192/29
IPv6:
2a01:4f8:2b03:38b::2/64
2a01:4f8:2b03:38c::2/64
2a01:4f8:2b03:38d::2/64
You can verify if a web crawler accessing your website is a legitimate DataForSEO Bot by running a reverse DNS lookup.
Reverse DNS mask: crawling-gateway-*.dataforseo.com
Example:
# host 136.243.228.176
176.228.243.136.in-addr.arpa domain name pointer crawling-gateway-136-243-228-176.dataforseo.com.
To disallow our bot on certain pages of your website, add the following snippet to your website’s robots.txt file:
User-agent: DataForSeoBot
Disallow: /[url path]
To disallow our bot on all pages of your website, add the following snippet to your website’s robots.txt file:
User-agent: DataForSeoBot
Disallow: /
Contact Us
