This update to the On-Page API brings greater clarity to your crawl results and improves reporting accuracy. We’ve added a new endpoint and expanded summary data, giving you more precise insight into the resources that don’t meet expected crawl conditions.
What’s new
This release adds a new Uncrawlable Resources endpoint to the On-Page API. It provides a dedicated list of resources detected on the target website that could not be crawled due to a content type inconsistency. This inconsistency means that the content type returned in the server response does not match the content type expected based on how the resource is referenced in the page HTML. When that happens, a resource is considered uncrawlable.
Besides a new dedicated endpoint, we’ve enriched the Summary endpoint with a new field in domain_info: total_uncrawlable_resources, reflecting the total number of such resources detected during the crawl.
How it works
Throughout the crawling process, our system evaluates each discovered resource against expected technical parameters, including its content type. When a mismatch is detected between the anticipated value and the returned value, the resource is flagged as uncrawlable.
The new Uncrawlable Resources endpoint lets you retrieve the full list of flagged URLs, while the Summary endpoint aggregates their total count for quick reporting and analysis.
Why it matters
By explicitly surfacing all uncrawlable resources, this update increases transparency and ensures that all crawl outcomes, including mismatches, are properly accounted for and accessible to you via API. This gives you improved debugging, clearer reporting, and more reliable technical SEO analysis.
Get started
Check the updated On-Page API documentation to learn more about the Uncrawlable Resources endpoint and the new total_uncrawlable_resources field in the Summary response.