How to scrape Google AI Overviews with SERP API
With the powerful Google Organic SERP API, you can track all possible SERP features, and AI Overviews are no exception. In this guide, we cover the key aspects of scraping Google AI Overviews with SERP API and the types of AI Overview data you receive.
Scraping Google AI Overviews with SERP API: key aspects
The AI Overviews automatically appear in the Google Organic SERP API responses whenever we detect them in the search results. You can locate structured AI Overview data in the dedicated ai_overview item in the API response’s items array. This way, you can easily identify when AI Overview is present.
Here’s an example of an ai_overview item respective to the AI Overview scraped from SERP:
{
"type": "ai_overview",
"rank_group": 1,
"rank_absolute": 1,
"page": 1,
"position": "left",
"xpath": "/html[1]/body[1]/div[3]/div[1]/div[13]/div[1]/div[2]/div[1]/div[1]/div[1]/div[1]/div[1]/div[1]",
"asynchronous_ai_overview": false,
"markdown": "\n\n\n\n\n\n[View all](https://www.google.com/search?q=classification+of+animals+vertebrates&sca_esv=cb427c73f925e7d3&hl=en&gl=GB&udm=2&source=univ&sa=X&ved=2ahUKEwi10pDcs5KRAxUWTTABHZClA-EQnN8JKAZ6BAgaEAo) \n\nVertebrates are classified into five main groups: `fish, amphibians, reptiles, birds, and mammals`.These groups are distinguished by key characteristics such as their body covering, how they regulate temperature (warm-blooded or cold-blooded), and where they live and lay eggs. \n[[1]](https://www.bbc.co.uk/bitesize/articles/zyt2jsg)[[2]](https://www.bbc.co.uk/bitesize/articles/zp6g7p3)[[3]](https://www.twinkl.co.uk/teaching-wiki/vertebrates#:~:text=Mammals%20-%20Mammals%20have%20warm%20blood,include%20frogs%2C%20toads%20and%20salamanders.)\n\n- **Fish:** Live in water, have fins and gills, and lay their eggs in water.\n- **Amphibians:** Live on both land and in water, have smooth skin, and lay their eggs in water.They are cold-blooded.Examples include frogs and salamanders.\n- **Reptiles:** Have scales and are cold-blooded, living on land and in water.They lay their eggs on land.Examples include lizards, snakes, and turtles.\n- **Birds:** Are warm-blooded, have feathers, beaks, and wings, and lay their eggs on land.\n- **Mammals:** Are warm-blooded, typically have hair or fur, and feed their young with milk.They are born alive.Examples include humans, cats, and lions.",
"items": [
{
"type": "ai_overview_element",
"position": "left",
"title": "Vertebrates are classified into five main groups: fish, amphibians, reptiles, birds, and mammals. These groups are distinguished by key characteristics such as their body covering, how they regulate temperature (warm-blooded or cold-blooded), and where they live and lay eggs.",
"text": "Fish: Live in water, have fins and gills, and lay their eggs in water. Amphibians: Live on both land and in water, have smooth skin, and lay their eggs in water. They are cold-blooded. Examples include frogs and salamanders. Reptiles: Have scales and are cold-blooded, living on land and in water. They lay their eggs on land. Examples include lizards, snakes, and turtles. Birds: Are warm-blooded, have feathers, beaks, and wings, and lay their eggs on land. Mammals: Are warm-blooded, typically have hair or fur, and feed their young with milk. They are born alive. Examples include humans, cats, and lions.",
"markdown": "- **Fish:** Live in water, have fins and gills, and lay their eggs in water.\n- **Amphibians:** Live on both land and in water, have smooth skin, and lay their eggs in water.They are cold-blooded.Examples include frogs and salamanders.\n- **Reptiles:** Have scales and are cold-blooded, living on land and in water.They lay their eggs on land.Examples include lizards, snakes, and turtles.\n- **Birds:** Are warm-blooded, have feathers, beaks, and wings, and lay their eggs on land.\n- **Mammals:** Are warm-blooded, typically have hair or fur, and feed their young with milk.They are born alive.Examples include humans, cats, and lions.",
"links": null,
"images": null,
"references": [
{
"type": "ai_overview_reference",
"position": "left",
"source": "BBC",
"domain": "www.bbc.co.uk",
"url": "https://www.bbc.co.uk/bitesize/articles/zyt2jsg",
"title": "How to classify plants and animals - KS2 Science curriculum - BBC",
"text": "Vertebrates can be then classified into five main groups: * Amphibians. * Birds. * Fish. * Mammals. * Reptiles."
},
{
"type": "ai_overview_reference",
"position": "left",
"source": "BBC",
"domain": "www.bbc.co.uk",
"url": "https://www.bbc.co.uk/bitesize/articles/zp6g7p3",
"title": "What is a vertebrate? - BBC Bitesize",
"text": "Vertebrates. Animals can be classified as either vertebrates or invertebrates. Vertebrates are animals that have a backbone inside..."
},
{
"type": "ai_overview_reference",
"position": "left",
"source": "Twinkl",
"domain": "www.twinkl.co.uk",
"url": "https://www.twinkl.co.uk/teaching-wiki/vertebrates#:~:text=Mammals%20-%20Mammals%20have%20warm%20blood,include%20frogs%2C%20toads%20and%20salamanders.",
"title": "What are Vertebrates? | Vertebrate Groups - Twinkl",
"text": "Mammals - Mammals have warm blood and usually have hair or fur on their bodies. Mammals are born alive and the mothers feed their ..."
}
]
}
],
"references": [
{
"type": "ai_overview_reference",
"position": "right",
"source": "BBC",
"domain": "www.bbc.co.uk",
"url": "https://www.bbc.co.uk/bitesize/articles/zyt2jsg",
"title": "How to classify plants and animals - KS2 Science curriculum - BBC",
"text": "Vertebrates can be then classified into five main groups: * Amphibians. * Birds. * Fish. * Mammals. * Reptiles."
},
{
"type": "ai_overview_reference",
"position": "right",
"source": "BBC",
"domain": "www.bbc.co.uk",
"url": "https://www.bbc.co.uk/bitesize/articles/zp6g7p3",
"title": "What is a vertebrate? - BBC Bitesize",
"text": "Vertebrates. Animals can be classified as either vertebrates or invertebrates. Vertebrates are animals that have a backbone inside..."
},
{
"type": "ai_overview_reference",
"position": "right",
"source": "Twinkl",
"domain": "www.twinkl.co.uk",
"url": "https://www.twinkl.co.uk/teaching-wiki/vertebrates#:~:text=Mammals%20-%20Mammals%20have%20warm%20blood,include%20frogs%2C%20toads%20and%20salamanders.",
"title": "What are Vertebrates? | Vertebrate Groups - Twinkl",
"text": "Mammals - Mammals have warm blood and usually have hair or fur on their bodies. Mammals are born alive and the mothers feed their ..."
}
],
"rectangle": null
},
The Google Organic SERP API supports the two AI Overview elements that are generated in search results:
➤ Synchronous (cached by Google) – this type of AI Overview is immediately available when the search results load. Google has already processed and cached the content, so it’s embedded directly within the HTML of the search results page. When you receive the results, this type of AI Overview is directly included in the response.
➤ Asynchronous – this AI Overview variant is not immediately available in the HTML content when the search results are first loaded. This occurs when the AI Overview requires additional time to generate or retrieve the necessary information, resulting in a slight delay before the overview becomes visible to the user.
However, the data from asynchronous AI Overviews is not retrieved by default. To get this data, set the load_async_ai_overview parameter to true. When set to true, the API will make an additional request to retrieve data if an asynchronous AI Overview is detected.
Note that your account will be additionally charged when this parameter is set to true in a request. For example, a Standard request with normal priority costs $0.0006. If this request contains "load_async_ai_overview": "true", you will be billed an additional $0.0006, resulting in a total charge per request of $0.0012. If the asynchronous AI Overview is not available in the SERP, the extra charge will be refunded. Learn more on the Pricing page.
Here are possible situations:
- If
"load_async_ai_overview": "true", but a cached AI Overview is present in the SERP, the additional charge is refunded. - If
"load_async_ai_overview": "true", and an asynchronous AI Overview is retrieved, the data is returned, and the additional charge applies. - If
"load_async_ai_overview": "false", and a cached AI Overview is present, the feature is returned without additional charge. - If
"load_async_ai_overview": "false", and an asynchronous AI Overview is detected, the API will return"ai_overview": null.
In the Standard HTML and Live HTML endpoints, you can set the expand_ai_overview parameter to true to “expand” the AI Overview element and get more detailed information with no additional charge. By default, this parameter is set to false. In the responses of Advanced SERP API endpoints, you will always get complete expanded JSON data from this feature.
Here’s what the expanded AI Overview may look like from the visualized HTML response:
Example of the additional parameters in the POST request payload:
[
{
"language_code": "en",
"location_code": 2840,
"keyword": "get organized at work tips",
"load_async_ai_overview": true,
"expand_ai_overview": true
}
]
Additionally, the Google Organic SERP API supports AI Overview elements, which may be generated within the following SERP features: Knowledge Graph, People Also Ask, and Product Considerations. The data for such AI Overview elements will appear in the respective items in the response: knowledge_graph_ai_overview_item, people_also_ask_ai_overview_expanded_element, and product_considerations_ai_overview_expanded_element. Check the Google Organic API documentation for more information.
What AI Overview data can you get?
With Google Organic, you can pull all important insights from AI Overviews. Let’s take a look at possible AI Overview components and the corresponding data elements in the API response.
1 AI-generated text summary and featured images. The AI-generated text summary and featured images are located in:
markdownfield of theai_overviewitem. The markdown field contains the whole summary text and links to all images formatted in the markdown markup language;- Separate
ai_overview_elementobjects with text summary, divided into segments, quoted websites, and featured images info.
Each of the ai_overview_element objects includes:
textfield that contains the part of the generated summary text;titlefield with title of element of the summary (if available);markdownfield with the element content formatted in markdown format;linksarray with links to featured websites and anchor text;imagesarray featuring alternative text, direct image links, and original source locations;referencesarray with additional details about the sources (posts, blog articles, or news pieces) used to generate the part of the summary. These details include the source name, domain, source URL, and extracted text segments.
2 List of cited sources. The detailed information about all the websites used to generate the AI Overview is located in the references array of the ai_overview item. The references array consists of individual ai_overview_reference objects that include:
sourcefield with the quoted website’s name;domainfield with the domain address of the website;urland title of the cited website;textfield containing the website text used to generate the response.
3 Featured video. The data on video or videos featured in the AI Overview can be found in a separate ai_overview_video_element.
In each ai_overview_video_element, you will find fields with the following information:
titleof the video element;snippetfield with additional information for the video;urlfield with the link to the video;domainfield containing domain of the website hosting the video;image_urlfield with the URL to the image thumbnail of the video;sourcefield with name of the video source;dateandtimestampwhen the video was posted.
4 AI-generated table. The AI Overview may occasionally include a table with structured information relevant to the query. If a table appears in the AI Overview, the information from it will be structured in the ai_overview_table_element of the response.
The ai_overview_table element includes:
markdownfield with the table content structured in markdown markup language;tableobject that contains thetable_headerandtable_contentarrays;table_headerarray with the content of the header of the table;table_contentarrays with the table content. Each array represents the table row;referencesarray with information about the sources used to generate the table.
As you can see, with the Google Organic SERP API, you can effortlessly retrieve all available AI Overview data in a structured format. From AI-generated text snippets and tables to lists of quoted sources, you will get the insights necessary for AI search optimization and more.
Additionally, we provide comprehensive data from the AI Mode – Google’s standalone AI-powered search engine. Learn more about the AI Mode in this blog post and fetch the data with our dedicated AI Mode SERP API.