How do I find content duplicates across the internet?
Using the new Content Duplicates endpoint of Backlinks API, you can easily find duplicates of any content across the internet. It will provide you with a list of pages that contain duplicate content from the page you specify in the POST request. You will obtain the URL of each page, its IP address, size in bytes, meta tag info, server info, and other relevant data.
How to use the endpoint
1 Specify the URL of the page you wish to find content duplicates for in the target field.
Note that you can specify only URLs in this field. When sending multiple requests simultaneously, the URLs in the target field must belong to the same domain to avoid errors.
2 Specify the content similarity score by adding the similarity parameter to the task POST body. You can set this score from 0 to 4. When set to 0, the API will return pages with content not similar (or minimally similar) to the content of the target page. When set to 4, the API will return pages with content highly similar to the content of the target page.
By default, this parameter is set to 2
3 Set the maximum number of pages you want to receive in the API response by adding the limit parameter.
Default limit value: 100
Maximum limit value: 1000
Your POST request should be structured as in the following example:
[
{
"target": "https://www.instructables.com/How-To-Start-A-Fire-1/",
"limit": 5,
"similarity": 2
}
]
Example of the API response:
{
"version": "0.1.20220819",
"status_code": 20000,
"status_message": "Ok.",
"time": "0.7417 sec.",
"cost": 0.02003,
"tasks_count": 1,
"tasks_error": 0,
"tasks": [
{
"id": "08311353-1535-0267-0000-d244fd35e8d0",
"status_code": 20000,
"status_message": "Ok.",
"time": "0.6664 sec.",
"cost": 0.02003,
"result_count": 1,
"path": [
"v3",
"backlinks",
"content_duplicates",
"live"
],
"data": {
"api": "backlinks",
"function": "content_duplicates",
"target": "https://www.instructables.com/How-To-Start-A-Fire-1/",
"limit": 5,
"similarity": 2
},
"result": [
{
"target": "https://www.instructables.com/How-To-Start-A-Fire-1/",
"similarity": 2,
"total_count": 1,
"items_count": 1,
"items": [
{
"type": "backlinks_content_duplicate",
"similarity": 4,
"main_domain": "garbagedisposal101.com",
"domain": "garbagedisposal101.com",
"tld": "com",
"page": "https://garbagedisposal101.com/garbage-disposal-near-me-denver-co.html",
"ip": "103.113.11.190",
"first_visited": "2019-02-23 19:24:47 +00:00",
"prev_visited": null,
"fetch_time": "2019-02-23 19:24:47 +00:00",
"status_code": 200,
"location": null,
"size": 11397,
"encoded_size": 11397,
"content_encoding": null,
"media_type": "text/html",
"server": "cloudflare",
"meta": {
"title": "garbage disposal plug Brighton Colorado 80602\n | garbage disposal installation kit Parker 80138",
"canonical": null,
"internal_links_count": 7,
"external_links_count": 0,
"images_count": 2,
"words_count": 1252,
"page_spam_score": 0,
"social_media_tags": null,
"h1": null,
"h2": null,
"h3": null,
"images_alt": null,
"powered_by": null,
"language": "en",
"charset": "utf-8",
"platform_type": [
"unknown"
],
"technologies": {
"cdn": "cloudflare"
}
}
}
]
}
]
}
]
}