Website crawl
POST/crawl
Scrapfly Crawler API: starts a crawl of a website from a starting URL (up to pagelimit pages, 1 to 1,000 on this listing) and returns immediately with a crawler UUID to poll for status, content, search and archives.
curl -X POST https://api.orthogonal.com/v1/run \
-H 'Authorization: Bearer orth_live_••••' \
-H 'Content-Type: application/json' \
-d '{
"api": "scrapfly",
"path": "/crawl",
"method": "POST",
"body": {
"url": "",
"page_limit": ,
"content_formats": [
""
],
"search":
}
}'
{
"success": true,
"data": {
"env": "LIVE",
"links": { … },
"status": "PENDING",
"project": "default",
"crawler_uuid": "639d3d97-8cca-4bc9-b37f-8bca8fae4a81"
}
}
A real response, recorded when Orthogonal checked this endpoint.
Changes with the parameters.
$0.000164per callUp to $5.72 depending on the options.
Body parameters
More from Scrapfly
- Crawl cancelPOST/crawl/{crawler_uuid}/cancelFree
- Crawl searchGET/crawl/{crawler_uuid}/search$0.000164 per call
- Web scrape headers onlyHEAD/scrape$0.000164 per call
- Crawl archive downloadGET/crawl/{crawler_uuid}/artifactFree
- Web scrape OPTIONS requestOPTIONS/scrape$0.000164 per call
- Crawled page contentGET/crawl/{crawler_uuid}/contentsFree
- Crawl structured extractionPOST/crawl/extract$0.000491 per call
- Crawl statusGET/crawl/{crawler_uuid}/statusFree
- Multi-crawl searchPOST/crawl/search$0.000164 per call
- Crawled URL listGET/crawl/{crawler_uuid}/urlsFree
- Multi-crawl question answeringPOST/crawl/prompt$0.000491 per call
- Web scrape with POST bodyPOST/scrape$0.000164 per call