How to crawl specific website directories in OnPage API
in
In the OnPage API Task POST endpoint, you can customize the request to crawl only a specific directory of a website. This is particularly useful for targeted audits of specific site sections without processing the entire domain. Besides, you’ll spend less money on analysis as you only crawl and pay for the pages that matter to your audit.
Below, we’ll explain how you can set up OnPage API crawling requests for specific directories.
Customizing OnPage API requests for specific directories
The custom_robots_txt allows you to set custom robots.txt directives for a crawler to follow. With this parameter, you can make the crawler scan a particular directory of a website and block other directories from scanning. To use this parameter effectively, you must format the robots.txt directives correctly and configure additional parameters to fetch URLs within your target directory.
Let’s review the two request options with custom_robots_txt for crawling specific directories:
1 Option 1: Using a specific sitemap
Request example:
[
{
"target": "dataforseo.com",
"custom_robots_txt": "User-agent: *\nDisallow: /\nAllow: /help-center/ \n Sitemap:https://dataforseo.com/help-centre-sitemap1.xml",
"allow_subdomains": true,
"max_crawl_pages": 2,
"robots_txt_merge_mode": "override",
"respect_sitemap": true
}
]
Here is how this request works:
1. The custom_robots_txt is set to block all directories, except /help-center/, and specifies an exact sitemap containing the Help Center URLs. To override the original robots.txt directives, the parameter robots_txt_merge_mode is set to override.
Specifying the directory sitemap URL in the custom_robots_txt is required because "robots_txt_merge_mode": "override" completely replaces the original robots.txt file. This means the crawler won’t know where to find the sitemap unless you explicitly specify it.
Note: the custom_robots_txt directives must be separated by \n (newline characters).
2. The “respect_sitemap”: true enables the crawler to fetch Help Center URLs from the specified sitemap;
3. The “allow_subdomains”: true allows the crawler to scan links on all subdomains that match the Allow rule;
4. The max_crawl_pages limits the maximum number of pages crawled.
As a result, the crawler only processes URLs from the specified sitemap that belong to the /help-center/ directory.
Note: instead of specifying the sitemap URL directly in custom_robots_txt, you can indicate it using the custom_sitemap parameter.
Example:
“custom_sitemap”: “https://dataforseo.com/help-centre-sitemap1.xml”
Below is the respective response example of the OnPage API Pages endpoint.
Response example:
{
"version": "0.1.20251112",
"status_code": 20000,
"status_message": "Ok.",
"time": "0.6505 sec.",
"cost": 0,
"tasks_count": 1,
"tasks_error": 0,
"tasks": [
{
"id": "11141444-1535-0216-0000-0f1e68a8e292",
"status_code": 20000,
"status_message": "Ok.",
"time": "0.6348 sec.",
"cost": 0,
"result_count": 1,
"path": [
"v3",
"on_page",
"pages"
],
"data": {
"api": "on_page",
"function": "pages",
"limit": 2,
"target": "dataforseo.com",
"custom_robots_txt": "User-agent: *\nDisallow: /\nAllow: /help-center/ \n Sitemap:https://dataforseo.com/help-centre-sitemap1.xml",
"allow_subdomains": true,
"max_crawl_pages": 2,
"robots_txt_merge_mode": "override",
"respect_sitemap": true
},
"result": [
{
"crawl_progress": "finished",
"crawl_status": {
"max_crawl_pages": 2,
"pages_in_queue": 0,
"pages_crawled": 2
},
"search_after_token": null,
"current_offset": 0,
"total_items_count": 2,
"items_count": 2,
"items": [
{
"resource_type": "html",
"status_code": 200,
"location": null,
"url": "https://dataforseo.com/help-center/craft-data-driven-content-strategies-in-hours-with-dataforseo-mcp-and-claude-by-daniel-schoester",
"meta": {
"title": "How to Create AI Content Strategy with Claude (Fast) - DataForSEO",
"charset": 65001,
"follow": true,
"generator": "WP Rocket 3.19.4",
"htags": {
"h4": [
"Products",
"Automation Platforms",
"AI / LLM Agents",
"About us",
"Step-by-Step Guide",
"1 Input raw customer information and determine the best keywords to target",
"2 Create ideas for strategic content pages that will drive traffic",
"3 Analyze SERP data to understand the competitive landscape and ranking opportunities",
"4 Create one end-to-end SEO optimized page idea, based on the research results",
"5 Generate a data-backed content brief to finalize your strategy",
"Wrap-Up",
"APIs",
"Databases",
"Solutions",
"Knowledge Base",
"Resources",
"Legal information",
"Payment Methods:"
],
"h1": [
"How to Create AI Content Strategy with DataForSEO & Claude (Fast), By Daniel Schoester"
],
"h2": [
"Embed DataForSeo widget on your website"
]
},
"description": "Most content strategists spend weeks gathering data, analyzing keywords, and planning topics. What if you could do that in hours?",
"favicon": "https://dataforseo.com/wp-content/uploads/2022/04/cropped-favicon_512-32x32.png",
"meta_keywords": "eretur",
"canonical": "https://dataforseo.com/help-center/craft-data-driven-content-strategies-in-hours-with-dataforseo-mcp-and-claude-by-daniel-schoester",
"internal_links_count": 139,
"external_links_count": 14,
"inbound_links_count": 2,
"images_count": 12,
"images_size": 0,
"scripts_count": 39,
"scripts_size": 0,
"stylesheets_count": 0,
"stylesheets_size": 0,
"title_length": 65,
"description_length": 129,
"render_blocking_scripts_count": 14,
"render_blocking_stylesheets_count": 0,
"cumulative_layout_shift": 0,
"meta_title": null,
"content": {
"plain_text_size": 9649,
"plain_text_rate": 0.047731173912828405,
"plain_text_word_count": 1514,
"automated_readability_index": 14.114521758275423,
"coleman_liau_readability_index": 13.528745046235137,
"dale_chall_readability_index": 8.941607332529244,
"flesch_kincaid_readability_index": 36.71760237780714,
"smog_readability_index": 15.060836070336702,
"description_to_content_consistency": 1,
"title_to_content_consistency": 1,
"meta_keywords_to_content_consistency": 0
},
"deprecated_tags": null,
"duplicate_meta_tags": [
"generator"
],
"spell": null,
"social_media_tags": {
"og:locale": "en_US",
"og:type": "article",
"og:title": "How to Create AI Content Strategy with Claude (Fast) - DataForSEO",
"og:description": "Most content strategists spend weeks gathering data, analyzing keywords, and planning topics. What if you could do that in hours?",
"og:url": "https://dataforseo.com/help-center/craft-data-driven-content-strategies-in-hours-with-dataforseo-mcp-and-claude-by-daniel-schoester",
"og:site_name": "DataForSEO",
"og:updated_time": "2025-08-18T10:56:54+03:00",
"og:image": "https://dataforseo.com/wp-content/uploads/2025/08/Keyword-Research.gif",
"og:image:secure_url": "https://dataforseo.com/wp-content/uploads/2025/08/Keyword-Research.gif",
"og:image:width": "1536",
"og:image:height": "998",
"og:image:alt": "Keyword Research",
"og:image:type": "image/gif",
"twitter:card": "summary_large_image",
"twitter:title": "How to Create AI Content Strategy with Claude (Fast) - DataForSEO",
"twitter:description": "Most content strategists spend weeks gathering data, analyzing keywords, and planning topics. What if you could do that in hours?",
"twitter:image": "https://dataforseo.com/wp-content/uploads/2025/08/Keyword-Research.gif"
}
},
"page_timing": {
"time_to_interactive": 38,
"dom_complete": 38,
"largest_contentful_paint": 0,
"first_input_delay": 0,
"connection_time": 7,
"time_to_secure_connection": 12,
"request_sent_time": 0,
"waiting_time": 0,
"download_time": 19,
"duration_time": 38,
"fetch_start": 0,
"fetch_end": 38
},
"onpage_score": 95.24,
"total_dom_size": 202214,
"custom_js_response": null,
"custom_js_client_exception": null,
"resource_errors": {
"errors": [
{
"line": 1887,
"column": 100,
"message": "The attribute has already been added.",
"status_code": 501
}
],
"warnings": [
{
"line": 3,
"column": 1,
"message": "Has node with more than 60 childs.",
"status_code": 1
},
{
"line": 0,
"column": 0,
"message": "Has more that 1500 nodes.",
"status_code": 2
}
]
},
"broken_resources": false,
"broken_links": false,
"duplicate_title": false,
"duplicate_description": false,
"duplicate_content": false,
"click_depth": 0,
"size": 202214,
"encoded_size": 0,
"total_transfer_size": 0,
"fetch_time": "2025-11-14 12:44:46 +00:00",
"cache_control": {
"cachable": true,
"ttl": 14400
},
"checks": {
"no_content_encoding": false,
"high_loading_time": false,
"from_sitemap": true,
"is_redirect": false,
"is_4xx_code": false,
"is_5xx_code": false,
"is_broken": false,
"is_www": false,
"is_https": true,
"is_http": false,
"high_waiting_time": false,
"has_micromarkup": false,
"has_micromarkup_errors": false,
"no_doctype": false,
"has_html_doctype": true,
"canonical": true,
"no_encoding_meta_tag": false,
"no_h1_tag": false,
"https_to_http_links": false,
"size_greater_than_3mb": false,
"meta_charset_consistency": true,
"has_meta_refresh_redirect": false,
"has_render_blocking_resources": true,
"low_content_rate": true,
"high_content_rate": false,
"low_character_count": false,
"high_character_count": false,
"small_page_size": false,
"large_page_size": false,
"low_readability_rate": false,
"irrelevant_description": false,
"irrelevant_title": false,
"irrelevant_meta_keywords": true,
"title_too_long": false,
"has_meta_title": false,
"title_too_short": false,
"deprecated_html_tags": false,
"duplicate_meta_tags": true,
"duplicate_title_tag": false,
"no_image_alt": true,
"no_image_title": true,
"no_description": false,
"no_title": false,
"no_favicon": false,
"seo_friendly_url": true,
"flash": false,
"frame": false,
"lorem_ipsum": false,
"seo_friendly_url_characters_check": true,
"seo_friendly_url_dynamic_check": true,
"seo_friendly_url_keywords_check": true,
"seo_friendly_url_relative_length_check": true,
"recursive_canonical": false,
"canonical_chain": false,
"canonical_to_redirect": false,
"canonical_to_broken": false,
"has_links_to_redirects": false,
"is_orphan_page": true,
"is_link_relation_conflict": false,
"redirect_chain": false
},
"content_encoding": "br",
"media_type": "text/html",
"server": "cloudflare",
"is_resource": false,
"url_length": 131,
"relative_url_length": 109,
"last_modified": {
"header": "2025-11-13 16:19:36 +00:00",
"sitemap": "2025-08-18 07:56:54 +00:00",
"meta_tag": null
}
},
{
"resource_type": "html",
"status_code": 200,
"location": null,
"url": "https://dataforseo.com/help-center/how-to-pull-ranked-keywords-from-ai-overviews",
"meta": {
"title": "How to pull ranked keywords from AI Overviews – DataForSEO",
"charset": 65001,
"follow": true,
"generator": "WP Rocket 3.19.4",
"htags": {
"h4": [
"Products",
"Automation Platforms",
"AI / LLM Agents",
"About us",
"Step-by-step guide",
"APIs",
"Databases",
"Solutions",
"Knowledge Base",
"Resources",
"Legal information",
"Payment Methods:"
],
"h1": [
"How to pull ranked keywords from AI Overviews"
],
"h2": [
"Embed DataForSeo widget on your website"
]
},
"description": "To track or pull ranked keywords from AI Overviews, you can use the DataForSEO Labs Ranked Keywords endpoint. This endpoint provides a list of keywords for",
"favicon": "https://dataforseo.com/wp-content/uploads/2022/04/cropped-favicon_512-32x32.png",
"meta_keywords": "eretur",
"canonical": "https://dataforseo.com/help-center/how-to-pull-ranked-keywords-from-ai-overviews",
"internal_links_count": 133,
"external_links_count": 11,
"inbound_links_count": 2,
"images_count": 7,
"images_size": 0,
"scripts_count": 39,
"scripts_size": 0,
"stylesheets_count": 0,
"stylesheets_size": 0,
"title_length": 58,
"description_length": 155,
"render_blocking_scripts_count": 14,
"render_blocking_stylesheets_count": 0,
"cumulative_layout_shift": 0,
"meta_title": null,
"content": {
"plain_text_size": 11159,
"plain_text_rate": 0.05396478433913813,
"plain_text_word_count": 1656,
"automated_readability_index": 11.503082259014455,
"coleman_liau_readability_index": 10.05307506053268,
"dale_chall_readability_index": 12.366741692297627,
"flesch_kincaid_readability_index": 56.880534978077364,
"smog_readability_index": 12.354978490361713,
"description_to_content_consistency": 1,
"title_to_content_consistency": 1,
"meta_keywords_to_content_consistency": 0
},
"deprecated_tags": null,
"duplicate_meta_tags": [
"generator"
],
"spell": null,
"social_media_tags": {
"og:locale": "en_US",
"og:type": "article",
"og:title": "How to pull ranked keywords from AI Overviews – DataForSEO",
"og:description": "To track or pull ranked keywords from AI Overviews, you can use the DataForSEO Labs Ranked Keywords endpoint. This endpoint provides a list of keywords for",
"og:url": "https://dataforseo.com/help-center/how-to-pull-ranked-keywords-from-ai-overviews",
"og:site_name": "DataForSEO",
"og:updated_time": "2025-08-20T15:55:07+03:00",
"twitter:card": "summary_large_image",
"twitter:title": "How to pull ranked keywords from AI Overviews – DataForSEO",
"twitter:description": "To track or pull ranked keywords from AI Overviews, you can use the DataForSEO Labs Ranked Keywords endpoint. This endpoint provides a list of keywords for"
}
},
"page_timing": {
"time_to_interactive": 51,
"dom_complete": 51,
"largest_contentful_paint": 0,
"first_input_delay": 0,
"connection_time": 6,
"time_to_secure_connection": 11,
"request_sent_time": 0,
"waiting_time": 0,
"download_time": 34,
"duration_time": 51,
"fetch_start": 0,
"fetch_end": 51
},
"onpage_score": 95.24,
"total_dom_size": 206810,
"custom_js_response": null,
"custom_js_client_exception": null,
"resource_errors": {
"errors": [
{
"line": 1489,
"column": 28,
"message": "No paragraph has been found within the local scope.",
"status_code": 501
},
{
"line": 1501,
"column": 4,
"message": "The tag has been closed unexpectedly.",
"status_code": 501
},
{
"line": 1503,
"column": 28,
"message": "No paragraph has been found within the local scope.",
"status_code": 501
},
{
"line": 1681,
"column": 74,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1681,
"column": 82,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1681,
"column": 88,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1681,
"column": 94,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1681,
"column": 104,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1681,
"column": 113,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1681,
"column": 122,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1681,
"column": 128,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1740,
"column": 72,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1740,
"column": 80,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1740,
"column": 86,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1740,
"column": 92,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1740,
"column": 102,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1740,
"column": 111,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1740,
"column": 120,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1740,
"column": 126,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1860,
"column": 78,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1860,
"column": 86,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1860,
"column": 92,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1860,
"column": 98,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1860,
"column": 108,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1860,
"column": 117,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1860,
"column": 126,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1860,
"column": 132,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1919,
"column": 76,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1919,
"column": 84,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1919,
"column": 90,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1919,
"column": 96,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1919,
"column": 106,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1919,
"column": 115,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1919,
"column": 124,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1919,
"column": 130,
"message": "Character reference is not terminated by a semicolon.",
"status_code": 501
},
{
"line": 1941,
"column": 4,
"message": "The tag has been closed unexpectedly.",
"status_code": 501
},
{
"line": 2289,
"column": 100,
"message": "The attribute has already been added.",
"status_code": 501
}
],
"warnings": [
{
"line": 3,
"column": 1,
"message": "Has node with more than 60 childs.",
"status_code": 1
},
{
"line": 0,
"column": 0,
"message": "Has more that 1500 nodes.",
"status_code": 2
}
]
},
"broken_resources": false,
"broken_links": false,
"duplicate_title": false,
"duplicate_description": false,
"duplicate_content": false,
"click_depth": 0,
"size": 206810,
"encoded_size": 0,
"total_transfer_size": 0,
"fetch_time": "2025-11-14 12:44:44 +00:00",
"cache_control": {
"cachable": true,
"ttl": 14400
},
"checks": {
"no_content_encoding": false,
"high_loading_time": false,
"from_sitemap": true,
"is_redirect": false,
"is_4xx_code": false,
"is_5xx_code": false,
"is_broken": false,
"is_www": false,
"is_https": true,
"is_http": false,
"high_waiting_time": false,
"has_micromarkup": false,
"has_micromarkup_errors": false,
"no_doctype": false,
"has_html_doctype": true,
"canonical": true,
"no_encoding_meta_tag": false,
"no_h1_tag": false,
"https_to_http_links": false,
"size_greater_than_3mb": false,
"meta_charset_consistency": true,
"has_meta_refresh_redirect": false,
"has_render_blocking_resources": true,
"low_content_rate": true,
"high_content_rate": false,
"low_character_count": false,
"high_character_count": false,
"small_page_size": false,
"large_page_size": false,
"low_readability_rate": false,
"irrelevant_description": false,
"irrelevant_title": false,
"irrelevant_meta_keywords": true,
"title_too_long": false,
"has_meta_title": false,
"title_too_short": false,
"deprecated_html_tags": false,
"duplicate_meta_tags": true,
"duplicate_title_tag": false,
"no_image_alt": true,
"no_image_title": true,
"no_description": false,
"no_title": false,
"no_favicon": false,
"seo_friendly_url": true,
"flash": false,
"frame": false,
"lorem_ipsum": false,
"seo_friendly_url_characters_check": true,
"seo_friendly_url_dynamic_check": true,
"seo_friendly_url_keywords_check": true,
"seo_friendly_url_relative_length_check": true,
"recursive_canonical": false,
"canonical_chain": false,
"canonical_to_redirect": false,
"canonical_to_broken": false,
"has_links_to_redirects": false,
"is_orphan_page": true,
"is_link_relation_conflict": false,
"redirect_chain": false
},
"content_encoding": "br",
"media_type": "text/html",
"server": "cloudflare",
"is_resource": false,
"url_length": 80,
"relative_url_length": 58,
"last_modified": {
"header": "2025-11-13 16:23:40 +00:00",
"sitemap": "2025-08-18 13:16:05 +00:00",
"meta_tag": null
}
}
]
}
]
}
]
}
2 Option 2: Using a start URL (without sitemap)
Request example:
[
{
"target": "dataforseo.com",
"custom_robots_txt": "User-agent: *\nDisallow: /\nAllow: /help-center/",
"allow_subdomains": true,
"max_crawl_pages": 5,
"robots_txt_merge_mode": "override",
"respect_sitemap": false,
"start_url": "https://dataforseo.com/help-center/"
}
]
The main difference in this case is that you don’t specify the sitemap URL and block sitemap access with the “respect_sitemap”: false. Instead, you indicate the start_url that points directly to the allowed directory. The crawler will start at the specified URL and discover links from there, following the ones that match the Allow rule in the custom_robots_txt.
Example response of the Pages endpoint:
{
"version": "0.1.20251112",
"status_code": 20000,
"status_message": "Ok.",
"time": "0.6811 sec.",
"cost": 0,
"tasks_count": 1,
"tasks_error": 0,
"tasks": [
{
"id": "11141436-1535-0216-0000-98bcf3e6b50c",
"status_code": 20000,
"status_message": "Ok.",
"time": "0.6311 sec.",
"cost": 0,
"result_count": 1,
"path": [
"v3",
"on_page",
"pages"
],
"data": {
"api": "on_page",
"function": "pages",
"limit": 2,
"target": "dataforseo.com",
"custom_robots_txt": "User-agent: *\nDisallow: /\nAllow: /help-center/",
"allow_subdomains": true,
"max_crawl_pages": 2,
"robots_txt_merge_mode": "override",
"respect_sitemap": false,
"start_url": "https://dataforseo.com/help-center/"
},
"result": [
{
"crawl_progress": "finished",
"crawl_status": {
"max_crawl_pages": 2,
"pages_in_queue": 0,
"pages_crawled": 2
},
"search_after_token": null,
"current_offset": 0,
"total_items_count": 2,
"items_count": 2,
"items": [
{
"resource_type": "html",
"status_code": 200,
"location": null,
"url": "https://dataforseo.com/help-center",
"meta": {
"title": "Help Center – DataForSEO",
"charset": 65001,
"follow": true,
"generator": "WP Rocket 3.19.4",
"htags": {
"h4": [
"Products",
"Automation Platforms",
"AI / LLM Agents",
"About us",
"How to Define Targets and Analyze Data in the LLM Mentions API",
"Using Cross Aggregated Metrics in LLM Mentions API for Side-by-Side Analysis",
"LLM Mentions API subscription explained",
"How to access search queries data from ChatGPT?",
"Using the “stop_crawl_on_match” parameter in SERP API",
"Google Trends API Limits and Restrictions",
"What’s a GID in Google Shopping API?",
"Using the “shoprs” parameter in Google Shopping Merchant API",
"SERP API pricing & depth update FAQ",
"Connect Gemini CLI to DataForSEO MCP Server",
"APIs",
"Databases",
"Solutions",
"Knowledge Base",
"Resources",
"Legal information",
"Payment Methods:"
],
"h2": [
"Embed DataForSeo widget on your website"
]
},
"description": "DataForSEO Help Center",
"favicon": "https://dataforseo.com/wp-content/uploads/2022/04/cropped-favicon_512-32x32.png",
"meta_keywords": "eretur",
"canonical": "https://dataforseo.com/help-center",
"internal_links_count": 145,
"external_links_count": 10,
"inbound_links_count": 3,
"images_count": 7,
"images_size": 0,
"scripts_count": 36,
"scripts_size": 0,
"stylesheets_count": 0,
"stylesheets_size": 0,
"title_length": 24,
"description_length": 22,
"render_blocking_scripts_count": 14,
"render_blocking_stylesheets_count": 0,
"cumulative_layout_shift": 0,
"meta_title": null,
"content": {
"plain_text_size": 666,
"plain_text_rate": 0.0031925143687114416,
"plain_text_word_count": 100,
"automated_readability_index": 54.004,
"coleman_liau_readability_index": 15.655999999999999,
"dale_chall_readability_index": 14.7498,
"flesch_kincaid_readability_index": -55.71299999999995,
"smog_readability_index": 30.526468669271875,
"description_to_content_consistency": 1,
"title_to_content_consistency": 1,
"meta_keywords_to_content_consistency": 0
},
"deprecated_tags": null,
"duplicate_meta_tags": [
"generator"
],
"spell": null,
"social_media_tags": {
"og:locale": "en_US",
"og:type": "article",
"og:title": "Help Center – DataForSEO",
"og:description": "DataForSEO Help Center",
"og:url": "https://dataforseo.com/help-center",
"og:site_name": "DataForSEO",
"og:updated_time": "2025-06-16T18:15:23+03:00",
"twitter:card": "summary_large_image",
"twitter:title": "Help Center – DataForSEO",
"twitter:description": "DataForSEO Help Center",
"twitter:label1": "Time to read",
"twitter:data1": "Less than a minute"
}
},
"page_timing": {
"time_to_interactive": 47,
"dom_complete": 47,
"largest_contentful_paint": 0,
"first_input_delay": 0,
"connection_time": 6,
"time_to_secure_connection": 11,
"request_sent_time": 0,
"waiting_time": 0,
"download_time": 30,
"duration_time": 47,
"fetch_start": 0,
"fetch_end": 47
},
"onpage_score": 87.27,
"total_dom_size": 208676,
"custom_js_response": null,
"custom_js_client_exception": null,
"resource_errors": {
"errors": [
{
"line": 2068,
"column": 100,
"message": "The attribute has already been added.",
"status_code": 501
},
{
"line": 2103,
"column": 1,
"message": "The body has been closed wrong.",
"status_code": 501
}
],
"warnings": [
{
"line": 3,
"column": 1,
"message": "Has node with more than 60 childs.",
"status_code": 1
},
{
"line": 0,
"column": 0,
"message": "Has more that 1500 nodes.",
"status_code": 2
}
]
},
"broken_resources": false,
"broken_links": false,
"duplicate_title": false,
"duplicate_description": false,
"duplicate_content": true,
"click_depth": 0,
"size": 208676,
"encoded_size": 0,
"total_transfer_size": 0,
"fetch_time": "2025-11-14 12:36:35 +00:00",
"cache_control": {
"cachable": true,
"ttl": 14400
},
"checks": {
"no_content_encoding": false,
"high_loading_time": false,
"from_sitemap": false,
"is_redirect": false,
"is_4xx_code": false,
"is_5xx_code": false,
"is_broken": false,
"is_www": false,
"is_https": true,
"is_http": false,
"high_waiting_time": false,
"has_micromarkup": false,
"has_micromarkup_errors": false,
"no_doctype": false,
"has_html_doctype": true,
"canonical": true,
"no_encoding_meta_tag": false,
"no_h1_tag": true,
"https_to_http_links": false,
"size_greater_than_3mb": false,
"meta_charset_consistency": true,
"has_meta_refresh_redirect": false,
"has_render_blocking_resources": true,
"low_content_rate": true,
"high_content_rate": false,
"low_character_count": true,
"high_character_count": false,
"small_page_size": false,
"large_page_size": false,
"low_readability_rate": true,
"irrelevant_description": false,
"irrelevant_title": false,
"irrelevant_meta_keywords": true,
"title_too_long": false,
"has_meta_title": false,
"title_too_short": true,
"deprecated_html_tags": false,
"duplicate_meta_tags": true,
"duplicate_title_tag": false,
"no_image_alt": true,
"no_image_title": true,
"no_description": false,
"no_title": false,
"no_favicon": false,
"seo_friendly_url": true,
"flash": false,
"frame": false,
"lorem_ipsum": false,
"seo_friendly_url_characters_check": true,
"seo_friendly_url_dynamic_check": true,
"seo_friendly_url_keywords_check": true,
"seo_friendly_url_relative_length_check": true,
"recursive_canonical": false,
"canonical_chain": false,
"canonical_to_redirect": false,
"canonical_to_broken": false,
"has_links_to_redirects": false,
"is_orphan_page": false,
"is_link_relation_conflict": false,
"redirect_chain": false
},
"content_encoding": "br",
"media_type": "text/html",
"server": "cloudflare",
"is_resource": false,
"url_length": 34,
"relative_url_length": 12,
"last_modified": {
"header": "2025-11-13 16:15:57 +00:00",
"sitemap": null,
"meta_tag": null
}
},
{
"resource_type": "html",
"status_code": 200,
"location": null,
"url": "https://dataforseo.com/help-center/category/getting-started",
"meta": {
"title": "Getting Started – DataForSEO",
"charset": 65001,
"follow": true,
"generator": "WP Rocket 3.19.4",
"htags": {
"h4": [
"Products",
"Automation Platforms",
"AI / LLM Agents",
"About us",
"What is llms.txt for DataForSEO Docs?",
"DataForSEO Sandbox – Best Practices",
"Best practices for handling live endpoints in DataForSEO API",
"Rules and limitations of “keyword” and “keywords” fields in DataForSEO APIs",
"Rate Limits and Request Limits for DataForSEO API",
"DataForSEO API Playground: a handy way to test APIs",
"How does your free unlimited trial work?",
"Using filters in DataForSEO APIs",
"How to use the Errors and Webhook Resend endpoints to resend pingbacks and postbacks",
"How to set multiple tasks in one API request?",
"APIs",
"Databases",
"Solutions",
"Knowledge Base",
"Resources",
"Legal information",
"Payment Methods:"
],
"h2": [
"Embed DataForSeo widget on your website"
]
},
"description": null,
"favicon": "https://dataforseo.com/wp-content/uploads/2022/04/cropped-favicon_512-32x32.png",
"meta_keywords": "eretur",
"canonical": "https://dataforseo.com/help-center/category/getting-started",
"internal_links_count": 139,
"external_links_count": 10,
"inbound_links_count": 3,
"images_count": 7,
"images_size": 0,
"scripts_count": 36,
"scripts_size": 0,
"stylesheets_count": 0,
"stylesheets_size": 0,
"title_length": 28,
"description_length": 0,
"render_blocking_scripts_count": 14,
"render_blocking_stylesheets_count": 0,
"cumulative_layout_shift": 0,
"meta_title": null,
"content": {
"plain_text_size": 684,
"plain_text_rate": 0.003613426733018479,
"plain_text_word_count": 102,
"automated_readability_index": 55.15176470588235,
"coleman_liau_readability_index": 15.84627450980392,
"dale_chall_readability_index": 14.811847058823531,
"flesch_kincaid_readability_index": -57.98205882352937,
"smog_readability_index": 30.526468669271875,
"description_to_content_consistency": null,
"title_to_content_consistency": 1,
"meta_keywords_to_content_consistency": 0
},
"deprecated_tags": null,
"duplicate_meta_tags": [
"generator"
],
"spell": null,
"social_media_tags": {
"og:locale": "en_US",
"og:type": "article",
"og:title": "Getting Started – DataForSEO",
"og:url": "https://dataforseo.com/help-center/category/getting-started",
"og:site_name": "DataForSEO",
"twitter:card": "summary_large_image",
"twitter:title": "Getting Started – DataForSEO",
"twitter:label1": "Help Сenter",
"twitter:data1": "19"
}
},
"page_timing": {
"time_to_interactive": 61,
"dom_complete": 61,
"largest_contentful_paint": 0,
"first_input_delay": 0,
"connection_time": 7,
"time_to_secure_connection": 20,
"request_sent_time": 0,
"waiting_time": 0,
"download_time": 34,
"duration_time": 61,
"fetch_start": 0,
"fetch_end": 61
},
"onpage_score": 85.45,
"total_dom_size": 189330,
"custom_js_response": null,
"custom_js_client_exception": null,
"resource_errors": {
"errors": [
{
"line": 1858,
"column": 100,
"message": "The attribute has already been added.",
"status_code": 501
}
],
"warnings": [
{
"line": 4,
"column": 1,
"message": "Has node with more than 60 childs.",
"status_code": 1
},
{
"line": 0,
"column": 0,
"message": "Has more that 1500 nodes.",
"status_code": 2
}
]
},
"broken_resources": false,
"broken_links": false,
"duplicate_title": false,
"duplicate_description": false,
"duplicate_content": true,
"click_depth": 1,
"size": 189330,
"encoded_size": 0,
"total_transfer_size": 0,
"fetch_time": "2025-11-14 12:36:37 +00:00",
"cache_control": {
"cachable": true,
"ttl": 14400
},
"checks": {
"no_content_encoding": false,
"high_loading_time": false,
"from_sitemap": false,
"is_redirect": false,
"is_4xx_code": false,
"is_5xx_code": false,
"is_broken": false,
"is_www": false,
"is_https": true,
"is_http": false,
"high_waiting_time": false,
"has_micromarkup": false,
"has_micromarkup_errors": false,
"no_doctype": false,
"has_html_doctype": true,
"canonical": true,
"no_encoding_meta_tag": false,
"no_h1_tag": true,
"https_to_http_links": false,
"size_greater_than_3mb": false,
"meta_charset_consistency": true,
"has_meta_refresh_redirect": false,
"has_render_blocking_resources": true,
"low_content_rate": true,
"high_content_rate": false,
"low_character_count": true,
"high_character_count": false,
"small_page_size": false,
"large_page_size": false,
"low_readability_rate": true,
"irrelevant_description": false,
"irrelevant_title": false,
"irrelevant_meta_keywords": true,
"title_too_long": false,
"has_meta_title": false,
"title_too_short": true,
"deprecated_html_tags": false,
"duplicate_meta_tags": true,
"duplicate_title_tag": false,
"no_image_alt": true,
"no_image_title": true,
"no_description": true,
"no_title": false,
"no_favicon": false,
"seo_friendly_url": true,
"flash": false,
"frame": false,
"lorem_ipsum": false,
"seo_friendly_url_characters_check": true,
"seo_friendly_url_dynamic_check": true,
"seo_friendly_url_keywords_check": true,
"seo_friendly_url_relative_length_check": true,
"recursive_canonical": false,
"canonical_chain": false,
"canonical_to_redirect": false,
"canonical_to_broken": false,
"has_links_to_redirects": false,
"is_orphan_page": false,
"is_link_relation_conflict": false,
"redirect_chain": false
},
"content_encoding": "br",
"media_type": "text/html",
"server": "cloudflare",
"is_resource": false,
"url_length": 59,
"relative_url_length": 37,
"last_modified": {
"header": "2025-11-13 19:04:08 +00:00",
"sitemap": null,
"meta_tag": null
}
}
]
}
]
}
]
}
This option is the better choice if your website doesn’t have separate sitemaps for each directory.
Using these request setups, you can easily crawl separate directories of your website and perform more focused technical audits. If you have any additional questions, don’t hesitate to contact our 24/7 customer support.