How to Build a Bulk Keyword Rank Checker API with DataForSEO

Say you track 10,000 keywords in five locations. That’s 50,000 SERPs every morning. If your own scraper collects them, you also pay for proxies, deal with captchas, and fix a parser each time Google changes its layout. That maintenance usually falls on developers who were hired to build the product. A bulk keyword rank checker API removes the collection layer from your stack: you send keywords and get structured positions back.

This tutorial builds on the DataForSEO SERP API. You’ll submit keywords in batches of 100, stop each crawl once your domain shows up using `stop_crawl_on_match`, collect results through `tasks_ready`, and pull out the positions you need.

What a bulk keyword rank checker needs to do

A rank checker answers one question many times over: where does this domain rank for this keyword, in this location, on this device? Each combination is its own SERP. Track 2,000 keywords on desktop and mobile in three countries, and one run is 12,000 SERPs.

Finding the domain and recording its position takes a few lines of code. The trouble is everything around it:

• Throughput. You need to send tens of thousands of queries without running into rate limits.
• Depth and cost. A site at position 74 needs eight SERP pages crawled. A site at position 3 needs one. If you pay for eight pages on every keyword, the bill grows quickly.
• Mapping. Results can arrive in any order, and each must land in the correct row in your database.
• Collection. You need to know when results are ready without hammering an endpoint.

Our SERP API covers each of these. Depth affects your unit economics the most, but first, you need to pick the mode that keeps bulk work affordable.

Live vs Standard mode: which one fits bulk rank checks

You can request Google organic results from our SERP API in two ways. Live mode returns the SERP in a single HTTP response, with one keyword per request.

Standard mode works with tasks: you post them, we process them in a queue, and you collect the results once they’re done.

Bulk rank checking fits Standard mode. The Task POST endpoint takes up to 100 tasks per call and up to 2,000 calls per minute, so one account can queue up to 200,000 keyword tasks a minute in theory. At normal priority, Standard mode also costs less per SERP than Live mode. Current rates are on the Google Organic SERP API pricing page. If a few keywords can’t wait, set `priority: 2` on those tasks. They run sooner and cost more.

Live mode is still useful when someone is waiting on the result, for example, a “check this keyword now” button in your app. A daily or weekly tracking run is a batch job, and nobody is watching each SERP come in.

The rest of this tutorial uses three endpoints:

1. Task POST submits up to 100 tasks per call.
2. Tasks Ready lists completed tasks you haven’t collected yet.
3. Task GET returns the full structured SERP for one task.

One parameter decides how much each of those tasks costs, so it’s worth covering before any code.

How stop_crawl_on_match cuts the cost of bulk rank checks

The SERP API returns the top 10 results by default. To find a domain further down, you raise `depth` (the maximum is 700). Billing is per SERP page of up to 10 results, so `depth: 100` on every keyword means paying for 10 pages each time, even when your domain sits at position 2.

`stop_crawl_on_match` solves that. You pass an array of up to 10 targets, and the crawl stops on the first SERP page where any result matches one of them. You pay for the pages crawled up to that point, and the depth you didn’t use isn’t charged. Our Help Center guide has an example: a task set to `depth: 100` found its target at position 7, and the charge went from $0.0155 to $0.002. How much you save depends on where your domains rank. Higher positions mean fewer pages and a smaller bill.

Each target has two required fields. `match_value` is the domain, subdomain, or URL pattern to look for, written without a protocol (for example, `example.com`). `match_type` sets how that value is matched:

1. `domain` matches that exact domain or subdomain and nothing else, so `blog.example.com` won’t match `shop.example.com`.
2. `with_subdomains` matches the root domain and every subdomain under it. For most rank trackers, this is the one to start with.
3. `wildcard` matches a URL pattern such as `/blog/post-*`. Use it when you only track one section of a site.

Three more parameters give you finer control, and each one only works when `stop_crawl_on_match` is set. `target_search_mode` decides whether the crawl stops when `any` target is found (the default) or waits until `all` of them are. `find_targets_in` restricts matching to certain SERP element types, such as `["organic"]`. `ignore_targets_in` does the opposite and skips types like `paid`. For the edge cases, see our guide to crawl control parameters.

Now, let’s put it in a request.

Step 1: Submit keywords in batches of 100

The snippet below takes keywords from your database, splits them into batches of 100, and posts each batch to Standard mode. Every task gets the same `stop_crawl_on_match` target, plus a `tag` with your internal keyword ID. When it finishes, each keyword has a task in the queue, and any rejected tasks are printed.

```python
import requests

API = "https://api.dataforseo.com/v3/serp/google/organic"
AUTH = ("your_login", "your_password")  # requests builds the Basic auth header from these

TARGETS = [{"match_type": "with_subdomains", "match_value": "example.com"}]

def build_task(kw):
    return {
        "keyword": kw["keyword"],
        "location_code": kw["location_code"],  # e.g. 2840 = United States
        "language_code": "en",
        "depth": 100,  # a ceiling, not a fixed cost: the crawl stops earlier on a match
        "stop_crawl_on_match": TARGETS,
        "find_targets_in": ["organic"],  # a paid ad for example.com shouldn't end the crawl
        "tag": str(kw["id"]),  # echoed back in the results so you can map SERPs to DB rows
    }

def submit(keywords):
    for i in range(0, len(keywords), 100):  # tasks above 100 per POST fail with error 40006
        batch = [build_task(kw) for kw in keywords[i:i + 100]]
        resp = requests.post(f"{API}/task_post", auth=AUTH, json=batch, timeout=60).json()
        for task in resp["tasks"]:
            if task["status_code"] != 20100:  # 20100 = "Task Created."
                print("rejected:", task["data"].get("tag"), task["status_message"])
```

See the Task POST docs.

Two details matter once this runs in production. `tag` holds up to 255 characters, so it can carry more than an ID if your schema needs it, for example `"8812|mobile|2826"`. And a batch call can succeed as a whole even if the individual tasks inside it fail. That’s why the loop checks the status of every task, not just the HTTP response.

With the tasks queued, the next job is getting them back.

Step 2: Collect results with tasks_ready or postbacks

You have two ways to find out if a task is done. Which one fits depends on volume.

The first is polling `tasks_ready`. The Tasks Ready endpoint returns the IDs of completed tasks you haven’t collected, up to 1,000 per call and up to 20 calls per minute. A task stays on that list for three days after it completes. The results stay available through `task_get` for 30 days, and you aren’t billed for collecting them.

The second is postbacks. Set `postback_url` (and `postback_data: "advanced"`) on each task, and we send the gzipped result to your server as soon as it’s ready. Our docs recommend postbacks once you collect more than 1,000 tasks a minute. If your server takes longer than 10 seconds to respond, the task goes back to the `tasks_ready` list, so keep a polling job running as a backup. Our pingbacks and postbacks guide covers the setup.

To keep the example short, the next snippet polls. It gets the list of ready tasks and fetches each one from Task GET. Then it keeps only the organic items from your domain and saves the best position under the keyword ID from `tag`.

```python
def is_target(domain, target="example.com"):
    # mirrors match_type "with_subdomains" without matching lookalikes like notexample.com
    return domain == target or domain.endswith("." + target)

def collect():
    ready = requests.get(f"{API}/tasks_ready", auth=AUTH, timeout=60).json()
    for item in ready["tasks"][0]["result"] or []:
        res = requests.get(f"{API}/task_get/advanced/{item['id']}", auth=AUTH, timeout=60).json()
        task = res["tasks"][0]
        if task["status_code"] != 20000 or not task["result"]:
            continue  # log and retry later; don't record a missing SERP as "not ranking"
        serp = task["result"][0]
        hits = [
            {"rank_group": i["rank_group"], "rank_absolute": i["rank_absolute"], "url": i["url"]}
            for i in serp["items"] or []
            if i["type"] == "organic" and is_target(i.get("domain") or "")
        ]
        # items are ordered by position, so hits[0] is the best ranking URL;
        # None means the domain wasn't found within the crawled depth
        save(task["data"]["tag"], hits[0] if hits else None, serp["check_url"])
```

See the Task GET Advanced docs.
`save()` is a placeholder for your own database write.

Run `collect()` in a loop or on a schedule until `tasks_ready` comes back empty. At 20 calls a minute, one worker can clear up to 20,000 task IDs a minute.

Step 3: Read rank_group, rank_absolute, and url

Here’s a trimmed version of what `collect()` receives for one keyword. It only shows the fields a rank checker needs and the items from the target domain. The values are made up for illustration. A real `items` array also holds every other organic result and SERP feature up to the page where the crawl stopped.

```json
{
  "keyword": "bulk keyword rank checker api",
  "location_code": 2840,
  "language_code": "en",
  "check_url": "https://www.google.com/search?q=bulk%20keyword%20rank%20checker%20api&num=100&hl=en&gl=US",
  "pages_count": 2,
  "items": [
    {
      "type": "organic",
      "rank_group": 12,
      "rank_absolute": 15,
      "domain": "example.com",
      "url": "https://example.com/rank-checker-api/"
    },
    {
      "type": "organic",
      "rank_group": 19,
      "rank_absolute": 24,
      "domain": "docs.example.com",
      "url": "https://docs.example.com/serp/rankings/"
    }
  ]
}
```

Note that `rank_group` and `rank_absolute` measure different things, and people often mix them up.

1. `rank_group` counts only elements of the same type. A `rank_group` of 12 means the 12th organic result, and that’s the number most rank trackers show as "position."
2. `rank_absolute` counts every element on the SERP, including featured snippets, People Also Ask boxes, local packs, and ads. In the example above, 12 vs 15 means three non-organic elements sit above the result. That gap can explain why clicks drop even when the organic position holds steady.

Store both numbers and `url`. If a keyword’s ranking URL changes from one day to the next, the usual causes are cannibalization or a redirect, and your users will want to know about it. `check_url` opens the exact Google query we crawled, so anyone on your team can spot-check a result.

`pages_count` shows how many SERP pages were crawled before the stop. Log it on every run, and you’ll see how much depth `stop_crawl_on_match` saves across your keyword set.

Step 4: Store, schedule, and diff positions daily

Once rankings come back reliably, the rest is standard backend work. A minimal schema stores one row per keyword per run: `keyword_id`, `checked_at`, `rank_group`, `rank_absolute`, `url`, `pages_count`. From that, you can chart position history, work out daily movement, and send an alert when a keyword drops past a threshold.

A common schedule runs `submit()` once a day at a set hour for each location, then runs `collect()` every minute until the queue is empty. Submission and collection run separately, so a slow batch doesn’t hold up the next one. If a worker crashes, it can resume from `tasks_ready`.

Teams moving off a scraper tend to notice the difference at this point. The pipeline needs no proxies, no captcha solving, and no HTML parsing, so your developers spend their time on product logic.

Common mistakes when checking keyword rankings in bulk

Skipping `tag`. Without it, you have to match results to keywords by comparing keyword strings, location codes, and device settings. That breaks on duplicates and on encoding differences.

When DataForSEO Labs API is a faster shortcut

All of the above assumes you already know which keywords to track. If you want to know what a domain ranks for right now, running SERP tasks won’t help, because you’d need the keywords first.

Our DataForSEO Labs API covers that case. In one call, it returns the keywords a domain already ranks for, with positions, from our own database. A practical setup is to pull a domain’s ranking keywords from Labs, then add the important ones to your daily SERP API checks to get fresh positions for each location. Our Help Center shows how to check Google keyword rankings in bulk with the Labs API.