max_crawl_pages is a new parameter added to SERP API and the Google Shopping Products and Amazon Products endpoints of Merchant API. In the following paragraphs, we explain what it is and how it works.
As you probably know, our APIs support the depth parameter, which indicates how many SERP results to return in the API response. When you set a task with a specific depth, our API crawls as many search engine result pages as necessary to provide the specified number of results.
Depending on a keyword and numerous other factors, a search engine might return a different number of results on its first page, and you would never guess what depth to set to receive the results from the first page only.
To tackle the issue, our dev team developed the max_crawl_pages parameter.
How it works
max_crawl_pages simply limits the number of pages to crawl.
Let us explain how it works with an example.
Suppose you sent a request to Google Organic SERP API with a depth of 15.
- The first page of Google contains 14 SERP results;
- The second page contains 10 results;
In this case, you would receive results from 2 search pages, paying 75% more for this request.
By contrast, if you sent the same request with ”max_crawl_pages”: 1, our API would crawl the first page of Google only, providing 14 SERP results in the response and minimizing the cost of SERP API.