How can I crawl a specific page? Can I have several pages scanned before the rest of the website?

DataForSEO On-Page API is a customizable crawling engine that can fit projects and tasks of any size.

If you need to scan only one page or prioritize several pages in a website crawl, we can offer you two options to do so. You can review them below, and choose the one that suits you better.

Option 1. Standard method.

All additional parameters available, results can be requested with any necessary endpoint.

1 To crawl one page using this option, you should set a task to On-Page API specifying a site in the target and the necessary URL in the start_url field. Importantly, you should limit the max_crawl_pages to 1.

Example:

POST: https://api.dataforseo.com/v3/on_page/task_post

[
  {
    "target": "dataforseo.com",
    "start_url": "https://dataforseo.com/apis",
    "max_crawl_pages": 1,
    "load_resources": true,
    "enable_javascript": true,
    "custom_js": "meta = {}; meta.url = document.URL; meta;",
    "tag": "some_string_123",
    "pingback_url": "https://your-server.com/pingscript?id=$id&tag=$tag"
  }
]

2 To crawl a list of several pages prior to other pages of a given domain, you should specify the necessary pages in the priority_urls array. Note that you can set up to 20 specific URLs of a single website.

If you want to scan the specified URLs only, you should additionally set the “max_crawl_pages” to the corresponding number of pages.

Example:

POST: https://api.dataforseo.com/v3/on_page/task_post

[
  {
    "target": "dataforseo.com",
    "max_crawl_pages": 10,
    "priority_urls": [
      "https://dataforseo.com/apis/serp-api",
      "https://dataforseo.com/apis/keyword-data-api",
      "https://dataforseo.com/apis/traffic-analytics-api",
      "https://dataforseo.com/apis/on-page-api",
      "https://dataforseo.com/apis/dataforseo-labs-api"
    ],
    "load_resources": true,
    "enable_javascript": true,
    "enable_browser_rendering": true
  }
]

Option 2. Live method.

Limited additional parameters available, but results will be provided directly in the response from the API server.

1 To scan one page with this option, you should set a task to the Instant Pages endpoint. Using this endpoint, you should indicate the target URL in the url field.

Example:

POST: https://api.dataforseo.com/v3/on_page/instant_pages

[
    {
        "url": "https://dataforseo.com/blog",
        "enable_javascript": true,
        "custom_js": "meta = {}; meta.url = document.URL; meta;"
    }
]

2 To scan a list of pages using the Instant Pages endpoint, you should set separate tasks and specify the necessary URL for each task. Using this endpoint, you can send up to 20 tasks with one POST request. Below you can see an example of setting five tasks, that is five URLs to crawl.

Example:

POST: https://api.dataforseo.com/v3/on_page/instant_pages

[
  {
    "url": "https://dataforseo.com/blog/"
  },
  {
    "url": "https://dataforseo.com/help-center/"
  },
  {
    "url": "https://dataforseo.com/pricing/"
  },
  {
    "url": "https://dataforseo.com/apis/"
  },
  {
    "url": "https://dataforseo.com/"
  }
]