Push the boundaries of what DataForSEO can do

Learn about the latest updates

Fill out my online form
Back

Crawl the list of URLs with OnPage API

August,19 2021 14:32:06

Improvement
OnPage API
Crawl the list of URLs with OnPage API

Our team is glad to announce a new feature in On-Page API!

We’ve added the field that allows you to scan up to 20 specific URLs of a single website.

The specified URLs will be crawled prior to other webpages of a given domain.

Thus, primarily you can scan a specific list of URLs you are interested in, rather than all the pages on the site in a row.  All it takes is to specify the priority_urls field in the following way:

[
    {
        "target": "dataforseo.com",
        "max_crawl_pages": 5,
        "priority_urls": [
           "https://dataforseo.com/apis/serp-api",
           "https://dataforseo.com/apis/keyword-data-api",
           "https://dataforseo.com/apis/traffic-analytics-api",
           "https://dataforseo.com/apis/on-page-api",
           "https://dataforseo.com/apis/dataforseo-labs-api"
           
        ]
    }
]

Feel free to check the documentation.

Learn more