Web Scraper API

The Web Scraper API provides web scraper services to retrieve HTML and plaintext data from any website URL.

To retrieve web page metadata and URL information, use the Webpage API instead.

/v1/webscraper GET

https://api.api-ninjas.com/v1/webscraper

Returns the HTML or plaintext data scraped from a given URL. Maximum size of data returned is 2MB.

Parameters

  • url  required

    URL to scrape.

  • text_only  optional

    Whether to only extract visible text (ignores HTML tags and metadata). Must be either true or false. Default is false.

  • user_agent  optional

    User-Agent string to use in the request header.

Headers

  • X-Api-Key  required

    API Key associated with your account.

Sample Request Live Demo!

url
https://api.api-ninjas.com/v1/webscraper?url=https://example.com

Headers

X-Api-KeyLog in or sign up to get your API Key

Sample Response

JSON
1 2 3 { "data": "<!doctype html>\n<html>\n<head>\n <title>Example Domain</title>\n\n <meta charset=\"utf-8\" />\n <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n <style type=\"text/css\">\n body {\n background-color: #f0f0f2;\n margin: 0;\n padding: 0;\n font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n \n }\n div {\n width: 600px;\n margin: 5em auto;\n padding: 2em;\n background-color: #fdfdff;\n border-radius: 0.5em;\n box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n }\n a:link, a:visited {\n color: #38488f;\n text-decoration: none;\n }\n @media (max-width: 700px) {\n div {\n margin: 0 auto;\n width: auto;\n }\n }\n </style> \n</head>\n\n<body>\n<div>\n <h1>Example Domain</h1>\n <p>This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.</p>\n <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n" }

Code Examples

1 2 3 4 5 6 7 8 import requests api_url = 'https://api.api-ninjas.com/v1/webscraper?url=https://example.com' response = requests.get(api_url, headers={'X-Api-Key': 'YOUR_API_KEY'}) if response.status_code == requests.codes.ok: print(response.text) else: print("Error:", response.status_code, response.text)

If your programming language is not listed in the Code Example above, you can still make API calls by using a HTTP request library written in your programming language and following the above documentation.