You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/docs/browser-rendering/rest-api/crawl-endpoint.mdx
+10-7Lines changed: 10 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,26 +38,29 @@ If you are on a Workers Free plan, your crawl may fail if it hits the [limit of
38
38
39
39
### Initiate the crawl job
40
40
41
-
Here is an example of how to initiate a crawl job with `url`, `limit`, `depth`, and `formats` parameters. The API will respond immediately with a job `id` you will use to retrieve the status and results of the crawl job.
41
+
Here are the basic parameters you can use to initiate your crawl job:
42
+
-`url` — (Required) Starts crawling from this URL
43
+
-`limit` — (Optional) Maximum number of pages to crawl (default is 10, maximum is 100,000)
44
+
-`depth` — (Optional) Maximum link depth to crawl from the starting URL
45
+
-`formats` — (Optional) Response format (default is HTML, other options are Markdown and JSON)
42
46
43
-
See the [advanced usage section below](/browser-rendering/rest-api/crawl-endpoint/#initiate-the-crawl-job) for additional parameters:
47
+
The API will respond immediately with a job`id` you will use to retrieve the status and results of the crawl job.
44
48
49
+
See the [advanced usage section below](/browser-rendering/rest-api/crawl-endpoint/#initiate-the-crawl-job) for additional parameters.
50
+
51
+
Here is an example that uses the basic parameters:
45
52
```bash
46
53
curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
// Optional: Maximum number of pages to crawl (default is 10, maximum is 100,000)
55
60
"limit": 50,
56
61
57
-
// Optional: Maximum link depth to crawl from the starting URL
58
62
"depth": 2,
59
63
60
-
// Optional: Response format (default is HTML, other options are Markdown and JSON)
61
64
"formats": ["markdown"]
62
65
}'
63
66
```
@@ -134,7 +137,7 @@ A successful cancellation will return a `200 OK` status code. The job status wil
134
137
135
138
## Advanced usage
136
139
137
-
The `/crawl` endpoint has many parameters you can use to customize your crawl. For the full list, check the [API docs[(https://developers.cloudflare.com/api/resources/browser_rendering/).
140
+
The `/crawl` endpoint has many parameters you can use to customize your crawl. For the full list, check the [API docs[(https://developers.cloudflare.com/api/resources/browser_rendering/).
138
141
139
142
Here is an example that uses the additional parameters that are currently available, in addition to the [basic parameters shown in the example above](/browser-rendering/rest-api/crawl-endpoint/#initiate-the-crawl-job) and the [render parameter below](/browser-rendering/rest-api/crawl-endpoint/#render-a-simple-html-fetch):
0 commit comments