We may earn a commission when you shop through links in this article or on our store pages, at no extra cost to you.
Two very different people type "web scraping API" into Google. One is building an AI assistant that needs to read a docs site, a rival's pricing page or this morning's news. The other wants 10,000 prices from a shop that does everything it can to keep bots out. The tool built for the first job, 
FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store →, has more than 190,000 stars on GitHub, yet it came last when Proxyway tested 11 scraping APIs on heavily protected sites. That isn't a contradiction. It's two jobs sharing one name, and a tool made for the other job is the quickest way to overpay or get blocked.
What is a web scraping API?
It's a service you send a web address to, and it sends back what's on the page, having done the fiddly parts: loading pages that only appear once JavaScript runs, rotating the IP addresses (proxies) requests come from, and retrying when a site pushes back. Older services return raw HTML. Newer ones return Markdown, plain text with headings that language models read well, or JSON, tidy fields such as price and title. Most can also crawl a whole site from one address.
Why does it matter now?
AI apps run on web pages, and websites are pushing back. On 1 July 2025 Cloudflare switched to blocking AI crawlers by default unless they pay publishers, and Thales's 2026 Bad Bot Report counted bots as more than 53% of all web traffic in 2025, 40% of it classed as malicious. The money has followed: FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store → raised a Ad$75 million Series B in September 2026 and says more than 1.5 million users build with it.
What does it cost, and what's the catch?
Everyone sells credits, and a credit means something different at each one. On FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store → and 
Context.devContext.devContext.dev, formerly Brand.dev, is an API that turns websites into clean Markdown, HTML, screenshots or structured JSON, and returns company data such as logos, colours and fonts.Buying guide · View store → an ordinary page costs one credit with JavaScript and proxies included (AdFirecrawl, Context.dev). On ScrapingBee the same page costs 5 credits if it needs JavaScript and 25 behind premium proxies, and Zyte sorts every site into five price tiers. Prices are in US dollars: Firecrawl adds local tax at checkout, and ScrapingBee's prices exclude VAT.
The catch is what you pay for when it fails. Firecrawl charges a credit for a page that comes back 403 Forbidden, because it hands you the block page (AdFirecrawl docs). ScrapingBee doesn't bill requests a site blocks, ZenRows says failed requests are never charged, Bright Data bills only successes, and Context.dev says most failures are free. On a site that turns you away half the time, that difference is the bill.
The short answer: for feeding an AI app ordinary websites, FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store → is the most complete toolkit: 1,000 free pages a month, then $19 a month for 5,000. Context.devContext.devContext.dev, formerly Brand.dev, is an API that turns websites into clean Markdown, HTML, screenshots or structured JSON, and returns company data such as logos, colours and fonts.Buying guide · View store → does the same core job for less per page ($19 a month for 7,500) and adds company and brand data on the same key. Developers happy to run their own server can have Crawl4AI for nothing. For protected shops, the classic APIs earn their keep: ScrapingBee for predictable credits from $49 a month, Zyte for the best unblocking in independent tests, and Bright Data for paying only for pages that arrive. More tools are on our developer tools page. Prices are from October 2026, in US dollars before VAT.
Feeding an AI or scraping at scale: which job is yours?
It's mostly about the sites. LLM-ready scraping means the words on lots of ordinary pages (docs, blogs, company sites, news), cleaned into Markdown for a model, where price per page and output quality matter most. Classic scraping means fields such as price and stock from a few big sites guarded by anti-bot systems, where the success rate decides everything. In Proxyway's test, run in October 2025, Shein, G2 and Hyatt defeated nearly half the APIs, and Proxyway judged FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store → "best suited for crawling the long-tail rather than individual protected targets".
John Watson Rooney, a scraping educator who works at Zyte (the views are his own, he says), argues that anti-bot protection is better and more widespread, headless browsers are easy to spot, and any AI tool promising to scrape any site is probably overselling.
Which side of that line you're on decides most of the rest, so I'd like to know what readers are scraping for.
The web scraping APIs side by side
From each pricing page in October 2026, on monthly billing, in US dollars before VAT.
| Tool | Best at | Free plan | Paid from | The catch |
|---|---|---|---|---|
| FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store → | AI apps, whole-site crawls | 1,000 credits a month | $19 for 5,000 pages | Blocked pages still cost a credit |
| Context.devContext.devContext.dev, formerly Brand.dev, is an API that turns websites into clean Markdown, HTML, screenshots or structured JSON, and returns company data such as logos, colours and fonts.Buying guide · View store → | AI apps plus company data | 1,000 credits a month | $19 for 7,500 pages | Hosted only; DPA on Enterprise |
| Crawl4AI | Free, self-hosted | Open source; cloud credit to start | Cloud packs from $10 | You run browsers and proxies |
| Jina Reader | One page to Markdown | 20 requests a minute, no key | Prepaid tokens | No crawler |
| ScrapingBee | Predictable credits at scale | 1,000 credits | $49 for 250,000 credits | Protected pages cost 25 to 75 credits |
| Zyte | The hardest sites | $5 credit for 30 days | Pay as you go | Price depends on each site's tier |
| Bright Data | Paying only for success | 5,000 requests a month | $1.50 per 1,000 | "Premium" sites cost more |
| ZenRows | A middle ground | 5,000 credits a month | $19 for 45,000 credits | Same multipliers as ScrapingBee |
| Apify | Ready-made scrapers | $5 credit a month | $19 plus usage | Every scraper prices differently |
What do 10,000 pages a month cost?
Our arithmetic from each pricing page in October 2026, monthly billing before VAT. "Plain" pages need no JavaScript; "JavaScript" pages must load in a browser; "protected" means premium proxies plus a browser.
| Tool | Plain | JavaScript | Protected |
|---|---|---|---|
| FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store → | $44 | $44 | $44 |
| Context.devContext.devContext.dev, formerly Brand.dev, is an API that turns websites into clean Markdown, HTML, screenshots or structured JSON, and returns company data such as logos, colours and fonts.Buying guide · View store → | $25.60 | $25.60 | $25.60 |
| Crawl4AI Cloud | $5 | $10 | $20 |
| Apify Website Content Crawler | About $2 | $5 to $50 | Plus proxies |
| ScrapingBee | $49 | $49 | $49 ($99 with stealth) |
| ZenRows | $19 | Up to $39 | $69 |
| Zyte | $1.30 to $12.70 | $10.10 to $160.80 | Set by the site's tier |
| Bright Data | $15 | $15 | $15, more on premium sites |
Firecrawl's figure is its $19 plan plus five $5 top-ups, Context.dev's its $19 plan plus three top-ups at $2.20, Crawl4AI Cloud's 0.5 to 2 credits a page at $0.001 a credit (Crawl4AI), and Apify's its own estimate per 1,000 pages (Apify).
A flat price isn't a success rate, though, and Proxyway found variable-priced APIs charged up to 100 times their base rate at their strongest settings. Structured JSON costs 5 credits a page on Firecrawl and Context.dev, which puts 10,000 pages on their $99 plans.
Firecrawl: the toolkit most AI builders start with
FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store → is here because it covers more of the AI job in one API than anything else: scrape a page to Markdown, JSON or a screenshot, crawl a site (it obeys robots.txt), map every URL on a domain, search the web and get the pages back, click through pages in a cloud browser, and watch pages for changes. An official MCP server plugs it into Claude, ChatGPT, Cursor and other AI tools. The core is open source under AGPL-3.0, though the default self-hosted setup leaves out its anti-bot engine, screenshots and page actions (AdFirecrawl docs).
Firecrawl claims 96% coverage on its own 1,000-page benchmark from January 2026 (AdFirecrawl). Proxyway noticed that it fails fast rather than retrying, which "offloads the burden of retrying to the user". Nate Herk's walkthrough shows the other side: Claude Code driving Firecrawl to pull 200 job listings into a spreadsheet for about 30 credits (his description carries a Firecrawl partner link).
Who it suits: developers feeding agents, chatbots and search tools with ordinary websites.
Less suited to: heavily protected shops, and anyone who needs data kept outside the US, where Firecrawl's servers are (AdFirecrawl privacy policy).
Price: free for 1,000 credits a month with no card; Hobby is $19 a month or $16 billed yearly for 5,000 credits, and Standard $99 ($83) for 100,000, with a data processing agreement from Standard up (AdFirecrawl).
Context.dev: the same core job for less per page, plus company data
Context.devContext.devContext.dev, formerly Brand.dev, is an API that turns websites into clean Markdown, HTML, screenshots or structured JSON, and returns company data such as logos, colours and fonts.Buying guide · View store →, formerly Brand.dev, matches Firecrawl's core (scrape to Markdown, HTML, screenshots or JSON, crawl up to 500 pages a call, batches of up to 25,000 addresses, site maps, search and monitors) for less per page on small plans (Context.dev). A standard scrape is one credit with JavaScript and proxies included, and batches charge only for pages that succeed. Its extra is company data from a domain: logos, colours, fonts and profiles for 10 credits a lookup.
Context.dev claims a 99.96% success rate, but it wasn't in Proxyway's test, so nobody independent has checked that. It's hosted only, its servers are in the US (Context.dev privacy notice), its terms say all purchases are non-refundable (Context.dev terms), and an MSA or DPA comes only with Enterprise. Its trust centre lists SOC 2 Type II.
Who it suits: small teams that want crawling and company data from one key at the lowest per-page price here.
Less suited to: anyone who wants to self-host, or needs a DPA on a self-serve plan.
Price: free for 1,000 credits a month with no card; Developer is $19 a month or $16 billed yearly for 7,500 credits, and Pro $99 ($83) for 125,000 (Context.dev).
Crawl4AI and Jina Reader: the free routes
Crawl4AI is an open-source Python library under the Apache 2.0 licence, with more than 85,000 GitHub stars, that runs a browser on your own machine, turns pages into Markdown and can extract JSON with any AI model you point it at. It costs nothing, but proxies, servers and upkeep are yours. Better Stack's tour shows its speed and lists the trade-offs: Python only, AI extraction needs an API key unless you run a local model, and frequent updates.
A hosted version, Crawl4AI Cloud, sells credit packs from $10, though it calls its prices a soft launch. Jina Reader is simpler still: put r.jina.ai/ in front of an address and the page comes back as Markdown, free at 20 requests a minute without a key, with 10 million free tokens on a new key. It reads one address per request, says it doesn't try to get past anti-bot systems, and has belonged to Elastic since October 2025.
ScrapingBee: classic scraping with credits you can predict
ScrapingBee has the easiest pricing to reason about for classic scraping, and it was one of four APIs that opened Proxyway's protected sites more than 80% of the time. A plain request is 1 credit, JavaScript 5, premium proxies 10 (25 with JavaScript) and stealth proxies 75, and an auto mode tries the cheapest first and charges only for the one that worked (ScrapingBee docs). It returns Markdown too. The French company joined Oxylabs' group in an eight-figure deal in June 2025 (Tech.eu).
Who it suits: price trackers and product researchers who want a fixed monthly bill.
Less suited to: the very hardest sites, where Proxyway found credit models "evidently not ideal" on cost.
Price: 1,000 free credits with no card, then $49 a month for 250,000 credits or $99 for 1 million, exclusive of VAT (ScrapingBee).
Zyte and Bright Data: when the site fights back
Zyte, which maintains the open-source framework Scrapy, topped Proxyway's 2025 unblocking test. You pay per 1,000 successful responses, from $0.13 for the simplest sites without a browser to $16.08 for the hardest with one, with lower rates from a $100 monthly commitment and $5 of free credit for 30 days. Proxyway's complaint is that Zyte's costs are hard to know in advance.
Bright Data, best known for its proxy network, charges its Web Unlocker only for successful requests: 5,000 free a month, then $1.50 per 1,000, with CAPTCHA solving, browser rendering and Markdown output included. Sites on its "premium domains" list cost more (Bright Data docs), and it says it vets every customer. It wasn't in Proxyway's latest test.
ZenRows and Apify
ZenRows uses the same credit ladder as ScrapingBee from $19 a month for 45,000 credits, never charges failed requests, and Proxyway rated it a strong all-rounder, though its plan's concurrency limit hurt at speed. Apify is a marketplace of 85,845 ready-made scrapers it calls Actors, plus its own Website Content Crawler, which turns sites into Markdown for AI; the free plan gives $5 of credit a month. Proxyway found Apify's dedicated scrapers failed on about half its targets.
When do you need a real browser?
When the data sits behind clicks, forms or a login, fetching a page isn't enough. Browserbase rents browsers to AI agents: one free hour, then $20 a month for 100 hours. Cloudflare's Browser Run gives 10 browser hours on the $5-a-month Workers Paid plan and $0.09 an hour after, but its crawler obeys robots.txt and doesn't get past bot protection (Cloudflare docs). FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store →'s Interact covers lighter cases from 2 credits a browser minute (AdFirecrawl docs).
Is web scraping legal in the UK?
Reading public pages isn't a crime in itself, but five sets of rules apply, and none of them care that the page was public. This is general information, not legal advice.
Hacking law. The Computer Misuse Act 1990 makes it an offence to access data knowing the access is unauthorised. Getting past logins, paywalls or blocks you weren't meant to pass is where scraping gets near it.
Copyright. The UK's text and data mining exception covers only non-commercial research. In March 2026 the government dropped a broad exception with an opt-out as its preferred plan, and its report says UK law gives AI developers less flexibility than countries with such exceptions. So copying pages wholesale for a commercial AI app generally needs a licence or another exception.
Database right. Taking a substantial part of a database, or repeatedly taking small parts, can infringe the UK database right, which catches the price lists and listings classic scrapers target.
Website terms. Ticking "I accept" usually binds you; terms merely linked at the foot of a page are more debatable, according to law firm Bristows. robots.txt isn't law, and its standard says it is "not a form of access authorization", but it's how publishers tell AI crawlers to stay away.
Personal data. Names, reviews and profiles bring in UK GDPR. The ICO and other regulators say publicly accessible personal information is still subject to data protection law, and collecting it by accident still counts as processing. For training generative AI, legitimate interests is the only realistic lawful basis, and developers must show scraping is necessary and pass a balancing test that "invisible processing" makes hard (ICO, December 2024). The ICO's £7.5 million fine on Clearview AI for scraping images of UK residents from the web is still in the courts (ICO).
US rulings, such as Meta v Bright Data in 2024, don't carry over. And a VAT-registered UK business buying from these overseas suppliers normally accounts for the 20% VAT itself under the reverse charge.
Questions people ask
Is Firecrawl free?
Yes, for 1,000 credits a month (about 1,000 pages) with no card, and the AGPL-licensed core can be self-hosted free with fewer features. Paid plans start at $19 a month, or $16 billed yearly.
What is the best free web scraping API?
For AI apps, FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store → and Context.devContext.devContext.dev, formerly Brand.dev, is an API that turns websites into clean Markdown, HTML, screenshots or structured JSON, and returns company data such as logos, colours and fonts.Buying guide · View store → both give 1,000 pages a month, and Crawl4AI is free if you host it. For classic scraping, Bright Data's Web Unlocker gives 5,000 requests a month and ZenRows 5,000 credits.
What's the difference between web scraping and an API?
A site's own API is a structured feed it offers on its terms; scraping reads the pages people see. Where a site offers its own API, that's usually the safer route.
Firecrawl or Crawl4AI?
Firecrawl for a hosted service with search, crawling and browser actions; Crawl4AI for free open-source code, with proxies and servers handled yourself.
What are the best Firecrawl alternatives?
Context.devContext.devContext.dev, formerly Brand.dev, is an API that turns websites into clean Markdown, HTML, screenshots or structured JSON, and returns company data such as logos, colours and fonts.Buying guide · View store → for a hosted service at a lower price per page, Crawl4AI and Jina Reader for free routes, and Apify's Website Content Crawler for ready-made crawling. If an agent only needs search results, Tavily and Exa are search APIs instead.
So which web scraping API should you use?
The job decides it before the price does. For feeding an AI app ordinary websites, FirecrawlFirecrawlFirecrawl is a web data API that turns websites into clean Markdown or structured JSON for AI agents, chatbots and data pipelines, with endpoints to search, scrape, crawl and click1 current offer · View store → is the fullest toolkit and Context.devContext.devContext.dev, formerly Brand.dev, is an API that turns websites into clean Markdown, HTML, screenshots or structured JSON, and returns company data such as logos, colours and fonts.Buying guide · View store → the cheaper per-page option with company data on top, and both let you test 1,000 pages free. For protected shops, a classic API that charges only for success, such as ScrapingBee, Zyte or Bright Data, will cost more per page and get more of them. And if the pages hold personal data, the UK GDPR questions come before the tool.
More options and prices are on our developer tools page. If the data is heading for a site you're building, our Framer vs Webflow comparison covers the builders, and our guide to AI note takers covers the ICO's view of AI tools that record people.
