Web scraping is one of the most requested capabilities for AI agents. Whether it's gathering research, monitoring competitors, or extracting structured data from websites, agents need reliable tools to interact with the web.
Today we're excited to highlight ScrapeGraphAI, a powerful AI-powered web scraping API now available on Orthogonal.
What is ScrapeGraphAI?
ScrapeGraphAI is an intelligent web scraping platform that uses AI to extract structured data from any website. Unlike traditional scrapers that break when websites change, ScrapeGraphAI adapts automatically using large language models to understand page structure.
With over 22,000 stars on GitHub and 40 million+ extracted webpages, it's one of the most trusted scraping solutions for the AI era.
Key Features
SmartScraper
Extract specific data from any webpage using natural language. Just describe what you want—product details, contact info, or any structured content—and get clean JSON back.
Markdownify
Convert any webpage into clean, well-formatted Markdown instantly. Perfect for feeding web content into your AI models, documentation, or knowledge bases.
SearchScraper
Search and analyze data across the entire web from a single prompt. Ideal for market research, competitor analysis, or tracking brand mentions.
SmartCrawler
Crawl and analyze entire websites with intelligent depth control. Essential for documentation analysis, site-wide data extraction, or competitive intelligence.
AgenticScraper
An AI agent that autonomously navigates and interacts with websites to complete complex tasks. Perfect for multi-step workflows, form filling, or extracting data behind interactions.
Using ScrapeGraphAI with Orthogonal
SmartScraper - Extract Structured Data
import Orthogonal from "@orth/sdk";
const orthogonal = new Orthogonal({
apiKey: process.env.ORTHOGONAL_API_KEY,
});
// Extract structured data from any webpage
const result = await orthogonal.run({
api: "scrapegraph",
path: "/v1/smartscraper",
body: {
website_url: "https://example-shop.com/products",
user_prompt: "Extract all product names, prices, and availability"
}
});
console.log(result.data);Markdownify - Convert to Markdown
// Convert any page to clean markdown
const result = await orthogonal.run({
api: "scrapegraph",
path: "/v1/markdownify",
body: {
website_url: "https://docs.example.com/api-reference"
}
});
// Feed to your LLM or save to knowledge base
console.log(result.data);SearchScraper - Search the Web
// Search the web and extract results
const result = await orthogonal.run({
api: "scrapegraph",
path: "/v1/searchscraper",
body: {
user_prompt: "Find the top 5 competitors to Notion and extract their pricing"
}
});
console.log(result.data);Using x402 Protocol
ScrapeGraphAI on Orthogonal also supports x402—an open protocol that enables native payments in HTTP. With x402, your agents can pay for API calls directly using USDC stablecoins, with no API keys required.
// Install: npm install x402-fetch viem
import { wrapFetchWithPayment } from "x402-fetch";
import { privateKeyToAccount } from "viem/accounts";
const privateKey = process.env.PRIVATE_KEY;
if (!privateKey) {
throw new Error("PRIVATE_KEY environment variable is required");
}
const account = privateKeyToAccount(
privateKey.startsWith("0x") ? privateKey : `0x${privateKey}`,
);
const fetchWithPayment = wrapFetchWithPayment(fetch, account);
// Make a paid request - payment is automatic
const url = "https://x402.orth.sh/scrapegraph/v1/smartscraper";
const response = await fetchWithPayment(url, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
website_url: "https://example-shop.com/products",
user_prompt: "Extract all product names and prices"
})
});
const data = await response.json();
console.log(data);x402 enables a new paradigm where AI agents can autonomously pay for services using stablecoins—no subscriptions, no API key management, just seamless agent-to-agent payments.
Why Agents Love ScrapeGraphAI
Traditional scraping tools were built for developers writing scripts. ScrapeGraphAI was built for the AI-native era:
- Natural language queries: Describe what you want, not how to find it
- Structured output: Get JSON, not HTML soup
- Self-healing scrapers: LLMs adapt when page layouts change
- Production-grade: Proxy rotation, anti-bot bypass, and JavaScript rendering built-in
Available Endpoints
ScrapeGraphAI offers several powerful endpoints through Orthogonal:
| Endpoint | Description |
|---|---|
| SmartScraper | AI-powered structured data extraction from single pages |
| SearchScraper | Search and scrape results from search engines |
| SmartCrawler | Crawl entire websites with depth control |
| Markdownify | Convert any webpage to clean Markdown |
| AgenticScraper | AI agent that navigates and interacts with websites |
| Scrape | Basic HTML content extraction |
| Sitemap | Extract and parse sitemap URLs |
Try It Today
Sign up for Orthogonal and get $10 free credits to try ScrapeGraphAI and dozens of other APIs. No API keys to manage, no accounts to create—just instant access to powerful tools for your agents.