By default, OnPage API runs several sitewide scans to check website health. However, these checkups sometimes cause trouble. For instance, when our crawler checks whether a website has a custom 404 page, it adds random symbols after the website’s URL to land on the page that doesn’t exist. However, website analytics tools still track such visits, giving website owners false traffic reports.
To prevent that, you can disable sitewide checks in OnPage API when setting a POST request. Simply add the disable_sitewide_checks parameter to your Task POST body and specify the checkups you want to disable. You can turn off the following tests:
• test_page_not_found — indicates whether a website has a custom 404 page;
• test_canonicalization — indicates whether pages on a target website have rel=canonical tags;
• test_https_redirect — indicates whether a website redirects http requests to the https version;
• test_directory_browsing — indicates whether a website doesn’t allow accessing a file directory without authentication.
Example: "disable_sitewide_checks": ["test_directory_browsing", "test_page_not_found"]
Your POST request should be structured as in the following example:
[
{
"target": "dataforseo.com",
"max_crawl_pages": 10,
"disable_sitewide_checks": ["test_directory_browsing", "test_page_not_found"]
}
]