We have an update to share! We’ve added an option that enables you to better control the crawling process with On-Page API.
Now you can specify subdomains disallowed for crawling by simply using the disallowed_subdomains parameter. For example:
[
{
"target": "site.com",
"max_crawl_pages": 10,
“disallowed_subdomains”: ["status.site.com", "docs.site.com"]
}
]
This update is particularly useful for websites with multiple subdomains. It allows you to focus on crawling and analyzing the main domain with its most important subdomains while ignoring less important ones.
Note: to use disallowed_subdomains, the allow_subdomains parameter should be set to true.
Don’t forget to check our docs to handle the new parameter correctly.