People want to get the necessary information faster, and websites have mere seconds to impress. In Google’s terms, the first impression a website makes is counted based on loading speed (Largest Contentful Paint, LCP), responsiveness (First Input Delay, FID), and visual stability of content (Cumulative Layout Shift, CLS). These benchmarks, also known as Core Web Vitals, along with other user experience metrics have become ranking signals as of June 2021.
Keeping our hands on the pulse of the industry, DataForSEO has incorporated a possibility to audit Core Web Vitals (CWV) with On-Page API.
In this article, we will explain in more detail what these new metrics are, how the measurements are taken, and how to swiftly empower your website audit solution with Core Web Vitals data. But first, we’re inviting you to explore the timeline of Google’s updates regarding user experience to get the full picture of where the search engine is heading.
Contents:
The blurry line between SEO and UX
Core Web Vitals (CWV) at their core
Measuring Core Web Vitals with On-Page API
The blurry line between SEO and UX
SEO is an ongoing process and so are the challenges associated with it. Throughout recent years, Google has been putting more and more emphasis on the experiences users encounter on the web – from security to content and design. Here are the most important points that concern on-page SEO. You can also view our full infographic here.
This year, Core Web Vitals (CWV) are added to mobile search signals. The update should roll out gradually between June and August following the official announcement on Search Central Blog.
Yet, Google’s algorithm is ever-evolving, and the upcoming page experience update is just one step on the search engine’s development path. Upon the announcement of the update roll-out timing, Google added the following:
“Our work is ongoing, which is why we plan to incorporate more page experience signals going forward and update them on a yearly basis.”
Clearly, Google will continue bringing user experience (UX) to the forefront. The line between optimizing for search engine bots and for users gets more blurry, and so does the line between SEO and UX.
Besides the fact that Google will factor in page experience metrics, the most important link connecting SEO and UX is the shared end goal – make a visitor come back and eventually convert.
Having said that, SEOs are already expanding their performance testing to elements impacting user experience. Sure enough, to do it efficiently, SEO experts need proper user-first website audit solutions.
As you can see, SEO software cannot afford to ignore the page experience update. To make sure your software stays in line with SEO’s requirements, you need to start measuring Core Web Vitals as soon as possible. In the following parts, we’ll give you all details on Core Web Vitals and will show you how to quickly add them to your tool using DataForSEO On-Page API.
Core Web Vitals at their core
Essentially, Core Web Vitals are a set of metrics that Google uses to evaluate the user experience of a certain page. As we’ve already mentioned, these metrics are components of a larger “page experience signals” set. Now, let’s take a closer look at the Core Web Vitals.
Page loading speed is measured by its Largest Contentful Paint or LCP
“The Largest Contentful Paint (LCP) metric reports the render time of the largest image or text block visible within the viewport, relative to when the page first started loading.” Google
Page responsiveness is measured by the First Input Delay or FID
“FID measures the time from when a user first interacts with a page (i.e. when they click a link, tap on a button, or use a custom, JavaScript-powered control) to the time when the browser is actually able to begin processing event handlers in response to that interaction.” Google
Visual stability of page content is measured by the Cumulative Layout Shift or CLS
“CLS is a measure of the largest burst of layout shift scores for every unexpected layout shift that occurs during the entire lifespan of a page.” Google
How are Core Web Vitals measured?
Core Web Vitals can be measured based on two types of data: lab or field. Lab data is collected in a controlled environment and provides reproducible results for debugging, while field data is based on anonymized real user measurement and is more useful for performance analytics.
A part of Google’s tools, namely Search Console, CrUX Report, and Web Vitals extension capture field data. PageSpeed Insights combines both field and lab data to display CWV metrics. At the same time, Chrome DevTools and Lighthouse are using lab data.
On the one hand, tools running on field data allow seeing page performance through the eyes of actual website visitors. On the other hand, they give little to no opportunity to observe gradual improvements as website owners work through the page issues. The CrUX Report shows CWV data as an aggregated 28-day average and the same goes for the Search Console. In other words, SEOs won’t see any changes sooner than in 28 days. The rest of Google’s tools, both those on field and lab data, give insights per one page at a time only.
In the end, the SEO community is left to seek help from paid website audit software to tweak and refine their websites and monitor changes at scale in real-time.
The next part will explain how we measure Core Web Vitals in DataForSEO, and how to incorporate them into your software with On-Page API.
Measuring Core Web Vitals with On-Page API
At DataForSEO, we test pages for Core Web Vitals in our own system with a simulated page load in a controlled set of conditions. In other words, we provide performance metrics based on lab data. As we’ve explained in the previous part, using lab data is generally considered the best option for rectifying website issues harmful to user experience and seeing immediate results.
DataForSEO On-Page API can empower your tool with all the necessary capabilities to give users comprehensive on-demand reports on up-to-date Core Web Vitals performance. What’s more, our On-Page API can supply you with a wealth of other metrics for quality site-wide and per-page audits.
But first things first, let’s review how to obtain the vital page experience metrics.
At square one, we need to set a task using On-Page API Task POST.
Besides specifying the target, we recommend limiting the number of pages to crawl. To receive Core Web Vitals, you should also set the following parameters to true: load_resources, enable_javascript, enable_browser_rendering. Your POST request body should be structured as in the example below:
[
{
"target": "dataforseo.com",
"max_crawl_pages": 10,
"load_resources": true,
"enable_javascript": true,
"enable_browser_rendering": true
}
]
Essentially, the last three parameters will instruct our crawler to:
load_resourcessuch as images, stylesheets, scripts, and broken resourcesenable_javascript, that is, render and execute JavaScript on the scanned pagesenable_browser_renderingfor Core Web Vitals measurement
Once you’ve set a task to On-Page API, you can obtain Core Web Vitals in the responses of several endpoints:
- Pages endpoint returns data on each scanned page
- Pages by Resource endpoint returns data on pages that contain the specified resource
- Duplicate Tags endpoint returns a list of pages that contain duplicate title or description tags
- Duplicate Content endpoint lists pages that have content similar to the specified page
All these endpoints indicate detailed performance metrics for each listed page, including meta tags checks, page timing, and over 60 other on-page check-ups.
Core Web Vitals will be provided in separate fields of two arrays. In particular, you will find Cumulative Layout Shift in the meta array:
{
"meta": {
"cumulative_layout_shift": 0.10156329069412295
}
}
First Input Delay and Largest Contentful Paint will be returned in the page_timing array:
{
"page_timing": {
"largest_contentful_paint": 1648.4750006347895,
"first_input_delay": 0.0550001859664917
}
}
In addition to LCP, FID, and CLS, DataForSEO On-Page API will help you to:
Effortlessly check if every page is on HTTPS
Instantaneously see pages with high loading time
Measure the Time to Interactive (TTI): the time from when the start of page loading to when it’s visually rendered, its initial scripts are loaded, and it can quickly respond to user input.
Capture waiting time or Time to First Byte (TTFB): the time between the client request to the server and when the first byte of data in the browser is received.
What’s more, you can give users plenty of opportunities to deeper investigate the issues slowing down their websites. In particular, you can incorporate the Waterfall endpoint to obtain detailed information about the page loading time, time to secure connection, and the time it takes to load page resources. Also, you can use the Resources endpoint to get a list of resources for the necessary pages, including the URL, size, and fetch timing for each resource.
All this data can be built into a single dashboard to give users a complete technical view of their website’s loading performance.
Below you can review a full response sample from the Pages endpoint of On-Page API with all vital metrics and more.
{
"version": "0.1.20210304",
"status_code": 20000,
"status_message": "Ok.",
"time": "1.6100 sec.",
"cost": 0,
"tasks_count": 1,
"tasks_error": 0,
"tasks": [
{
"id": "03181244-1535-0216-0000-f448f1a7d5b5",
"status_code": 20000,
"status_message": "Ok.",
"time": "1.5067 sec.",
"cost": 0,
"result_count": 1,
"path": [
"v3",
"on_page",
"pages"
],
"data": {
"api": "on_page",
"function": "pages",
"limit": 1
},
"result": [
{
"crawl_progress": "finished",
"total_items_count": 25,
"items_count": 10,
"items": [
{
"resource_type": "html",
"status_code": 200,
"location": null,
"url": "https://dataforseo.com/",
"meta": {
"title": "SEO software API. API for agencies. Best SEO data API provider.",
"charset": 65001,
"follow": true,
"generator": "Powered by Slider Revolution 5.2.6 - responsive, Mobile-Friendly Slider Plugin for WordPress with comfortable drag and drop interface.",
"htags": {
"h4": [
"Products",
"By industry",
"By company type",
"Docs",
"Resources",
"Login",
"We will assist you during the whole process of implementation. We can make Skype call or group chat. Any communication channel that will satisfy your needs.",
"APIs",
"About",
"Legal information",
"Follow us on:",
"Payment Methods:"
],
"h2": [
"Our APIs",
"Need Any Help? Our 5-Star Support Will Help You.",
"Benefits",
"Insights",
"DataForSEO – the best data provider for SEO-software companies and agencies",
"Reviews",
"DataForSEO in Figures",
"Questions & Answers",
"Our happy customers"
],
"h5": [
"Simple pricing",
"Tech Support",
"Clear stats",
"High speed",
"Availability",
"Accuracy and freshness",
"Predictable pricing",
"Introduction to APIs for SEO software",
"Why You Must Build Your Own SEO Software (If You Haven’t Yet)",
"Development of SEO Tools. Where to Start?"
]
},
"description": "DataForSEO ➤➤➤ SEO Software API ➤➤➤ SEO API data Provider built for SEO-Software companies and agencies. ✓✓✓ Great Speed, Clear Stats, Simple Pricing. Try for free now!",
"favicon": "https://dataforseo.com/wp-content/uploads/2016/11/cropped-Favicon_512-32x32.png",
"meta_keywords": null,
"canonical": "https://dataforseo.com/",
"internal_links_count": 67,
"external_links_count": 42,
"inbound_links_count": 25,
"images_count": 86,
"images_size": 1973837,
"scripts_count": 39,
"scripts_size": 485966,
"stylesheets_count": 24,
"stylesheets_size": 182515,
"title_length": 63,
"description_length": 168,
"render_blocking_scripts_count": 19,
"render_blocking_stylesheets_count": 16,
"cumulative_layout_shift": 0.10156329069412295,
"content": {
"plain_text_size": 8816,
"plain_text_rate": 0.04412235746316464,
"plain_text_word_count": 1496,
"automated_readability_index": 10.186513088400346,
"coleman_liau_readability_index": null,
"dale_chall_readability_index": 7.586710847032416,
"flesch_kincaid_readability_index": 51.60166369578138,
"smog_readability_index": 19.086457156102238,
"description_to_content_consistency": 0.8500000238418579,
"title_to_content_consistency": 1,
"meta_keywords_to_content_consistency": null
},
"deprecated_tags": null,
"duplicate_meta_tags": [
"generator"
],
"spell": null,
"broken_html": {
"errors": null,
"warnings": [
{
"line": 0,
"message": "Has node with more than 60 childs."
},
{
"line": 0,
"message": "Has more that 1500 nodes."
}
]
},
"social_media_tags": {
"og:type": "article",
"og:title": "SEO software API. Best SEO data API provider.",
"og:description": "DataForSEO ➤➤➤ SEO Software API ➤➤➤ SEO API data Provider built for SEO-Software companies and agencies. ✓✓✓ Great Speed, Clear Stats, Simple Pricing. Try for free now!",
"og:url": "https://dataforseo.com/",
"og:image": "https://dataforseo.com/wp-content/uploads/2018/02/logo.jpg",
"article:published_time": "2016-07-21",
"article:modified_time": "2021-01-19",
"og:site_name": "API for SEO-software companies and agencies",
"fb:app_id": "979971002139459",
"twitter:card": "summary"
}
},
"page_timing": {
"time_to_interactive": 5809,
"dom_complete": 11235,
"largest_contentful_paint": 1648.4750006347895,
"first_input_delay": 0.0550001859664917,
"connection_time": 4,
"time_to_secure_connection": 9,
"request_sent_time": 0,
"waiting_time": 337,
"download_time": 4,
"duration_time": 354,
"fetch_start": 0,
"fetch_end": 354
},
"onpage_score": 100,
"total_dom_size": 4995678,
"custom_js_response": null,
"resource_errors": {
"errors": null,
"warnings": [
{
"line": 0,
"message": "Has node with more than 60 childs.",
"status_code": 1
},
{
"line": 0,
"message": "Has more that 1500 nodes.",
"status_code": 2
}
]
},
"broken_resources": false,
"broken_links": false,
"duplicate_title": false,
"duplicate_description": false,
"duplicate_content": false,
"click_depth": 0,
"size": 199938,
"encoded_size": 28507,
"total_transfer_size": 29011,
"fetch_time": "2021-03-18 12:44:10 +00:00",
"cache_control": {
"cachable": false,
"ttl": 0
},
"checks": {
"no_content_encoding": false,
"high_loading_time": false,
"is_redirect": false,
"is_4xx_code": false,
"is_5xx_code": false,
"is_broken": false,
"is_www": false,
"is_https": true,
"is_http": false,
"high_waiting_time": false,
"no_doctype": false,
"canonical": true,
"no_encoding_meta_tag": false,
"no_h1_tag": true,
"https_to_http_links": false,
"has_html_doctype": true,
"size_greater_than_3mb": false,
"meta_charset_consistency": true,
"has_meta_refresh_redirect": false,
"has_render_blocking_resources": true,
"redirect_chain": false,
"low_content_rate": true,
"high_content_rate": false,
"low_character_count": false,
"high_character_count": false,
"small_page_size": false,
"large_page_size": false,
"low_readability_rate": false,
"irrelevant_description": false,
"irrelevant_title": false,
"irrelevant_meta_keywords": false,
"title_too_long": false,
"title_too_short": false,
"deprecated_html_tags": false,
"duplicate_meta_tags": true,
"duplicate_title_tag": false,
"no_image_alt": true,
"no_image_title": true,
"no_description": false,
"no_title": false,
"no_favicon": false,
"seo_friendly_url": true,
"flash": false,
"frame": false,
"lorem_ipsum": false,
"has_misspelling": true,
"seo_friendly_url_characters_check": true,
"seo_friendly_url_dynamic_check": true,
"seo_friendly_url_keywords_check": true,
"seo_friendly_url_relative_length_check": true,
"is_orphan_page": false,
"is_link_relation_conflict": false,
"has_links_to_redirects": false,
"recursive_canonical": false,
"canonical_chain": false,
"canonical_to_redirect": false,
"canonical_to_broken": false
},
"content_encoding": "gzip",
"media_type": "text/html",
"server": "nginx/1.10.1 (Ubuntu)",
"is_resource": false
}
]
}
]
}
]
}
To learn more about the cost of tasks with the enable_browser_rendering parameter, visit this article on our Help Center.
Conclusion
A website audit is a tedious process, which requires diving into many technicalities. Besides that, search engines never sleep and never stop setting new requirements.
With the page experience update, UX and Core Web Vitals are put at the forefront of technical SEO, and more importantly, have become vital ranking factors.
Google once again reminds us it’s taken a user-centric approach. So, the next logical step for on-page SEO tools would be embracing the same perspective and adapting the functionality to it. With additional user-focused components in your software, you will be able to capture a larger share of customers and aim for a higher pricing level.
Here at DataForSEO, we can help you to implement sweeping website audit features fast and hassle-free. Following a simple integration process, our On-Page API will empower your tool with:
- Up-to-date Core Web Vitals
- A deep dive into the page loading speed
- Detailed performance check-ups for site architecture, HTML elements, and content quality