Web Scraper API

The Web Scraper API provides web scraper services to retrieve HTML and plaintext data from any website URL.

/v1/webscraper GET

https://api.api-ninjas.com/v1/webscraper

Returns the HTML or plaintext data scraped from a given URL. Maximum size of data returned is 2MB.

Parameters

  • url  required

    URL to scrape.

  • text_only  optional

    Whether to only extract visible text (ignores HTML tags and metadata). Must be either true or false. Default is false.

  • user_agent  optional

    User-Agent string to use in the request header.

Headers

  • X-Api-Key  required

    API Key associated with your account.

Sample Request Live Demo!

url
https://api.api-ninjas.com/v1/webscraper?url=https://example.com

Sample Response

JSON
1 2 3 { "data": "<!doctype html>\n<html>\n<head>\n <title>Example Domain</title>\n\n <meta charset=\"utf-8\" />\n <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n <style type=\"text/css\">\n body {\n background-color: #f0f0f2;\n margin: 0;\n padding: 0;\n font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n \n }\n div {\n width: 600px;\n margin: 5em auto;\n padding: 2em;\n background-color: #fdfdff;\n border-radius: 0.5em;\n box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n }\n a:link, a:visited {\n color: #38488f;\n text-decoration: none;\n }\n @media (max-width: 700px) {\n div {\n margin: 0 auto;\n width: auto;\n }\n }\n </style> \n</head>\n\n<body>\n<div>\n <h1>Example Domain</h1>\n <p>This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.</p>\n <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n" }

Code Examples

1 2 3 4 5 6 7 8 import requests api_url = 'https://api.api-ninjas.com/v1/webscraper?url=https://example.com' response = requests.get(api_url, headers={'X-Api-Key': 'YOUR_API_KEY'}) if response.status_code == requests.codes.ok: print(response.text) else: print("Error:", response.status_code, response.text)

If your programming language is not listed in the Code Example above, you can still make API calls by using a HTTP request library written in your programming language and following the above documentation.