NewConnect Claude, Gemini, ChatGPT, and other AI agents to API Ninjas via our MCP server →

Web Scraper API

The Web Scraper API provides web scraper services to retrieve HTML and plaintext data from any website URL.

To retrieve web page metadata and URL information, use the Webpage API instead.

(4.3)

From 2,209 users

500+

Applications using this API

API Status

Online - All Systems Operational

API Response Times

Average
P50
P75
P90
P95
325ms355ms379ms633ms1082ms

Didn't find what you were looking for? Suggest an improvement

/v1/webscraper GET

https://api.api-ninjas.com/v1/webscraper

Returns the HTML or plaintext data scraped from a given URL. Maximum size of data returned is 2MB.

Parameters

  • url  required

    URL to scrape.

  • text_only  optional

    Whether to only extract visible text (ignores HTML tags and metadata). Must be either true or false. Default is false.

  • user_agent  optional

    User-Agent string to use in the request header.

Headers

  • X-Api-Key  required

    API Key associated with your account.

Response

A JSON object with the following fields or an error if the request is unsuccessful.

  • data

    The scraped HTML or extracted text, depending on the text_only parameter.

The API may also include additional metadata fields in the future; refer to the sample response for the exact structure.

Sample Request Live Demo!

url
text_only
user_agent

Try our APIs in the API playground

Sign up for a free API key to get started — no credit card required.

https://api.api-ninjas.com/v1/webscraper?url=https%3A%2F%2Fexample.com

Headers

X-Api-KeyLog in or sign up to get your API Key

Sample Response

JSON
{
    data:"<!doctype html> <html> <head> <title>Example Domain</title> <meta charset="utf-8" /> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style type="text/css"> body { background-color: #f0f0f2; margin: 0; padding: 0; font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; } div { width: 600px; margin: 5em auto; padding: 2em; background-color: #fdfdff; border-radius: 0.5em; box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); } a:link, a:visited { color: #38488f; text-decoration: none; } @media (max-width: 700px) { div { margin: 0 auto; width: auto; } } </style> </head> <body> <div> <h1>Example Domain</h1> <p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p> <p><a href="https://www.iana.org/domains/example">More information...</a></p> </div> </body> </html> "
}

Code Examples

1 2 curl -X GET "https://api.api-ninjas.com/v1/webscraper?url=https://example.com" \ -H "X-Api-Key: YOUR_API_KEY"

If your programming language is not listed in the Code Example above, you can still make API calls by using a HTTP request library written in your programming language and following the above documentation.

Frequently Asked Questions

  • Yes, but you must have a premium subscription. Commercial use of the Web Scraper API is not permitted on the free tier. See our pricing page to choose a plan that fits your needs.
  • Sign up for a free account to instantly get your API key, then pass it in the X-Api-Key header on every request. The documentation above includes ready-to-run examples in cURL, Python, JavaScript, Node.js, and more.
  • Your API requests will simply be denied once you hit your monthly quota — you will never be charged more than the plan you signed up for. To increase your quota, you can upgrade your plan any time on our pricing page.
  • The API responds with a standard HTTP status code and a JSON error message describing the problem. See our error codes reference for the full list of codes and how to resolve each one, or contact support if you need help.
  • The Web Scraper API retrieves the raw HTML or visible plaintext of any website URL, making it useful for content monitoring, data extraction, archiving, and feeding text into downstream tools. If you instead need structured page metadata such as title, description, and links, use the Webpage API.
  • Set the optional text_only parameter to true, which strips HTML tags and metadata and returns only the visible text; it defaults to false, returning the full HTML. You can try both modes in the live demo on the API documentation page.
  • The response is a JSON object with a data field containing the scraped HTML or extracted text depending on the text_only parameter, with a maximum returned size of 2MB. If a request is unsuccessful you will receive an error instead.
  • Yes, pass the optional user_agent parameter to set the User-Agent string sent in the request header, while the required url parameter specifies the page to scrape. For extracting text from images rather than web pages, see the Image to Text API.