Crawl

Crawl — see the properties below.

This node is also exposed as an AI Agent tool.

This node has 1 input port and 1 output port.

Common Properties

Every node shares these.

Name — the node's display name on the canvas.
Color — the node's colour on the canvas.
Delay Before (sec) — wait this long before the node runs.
Delay After (sec) — wait this long after the node runs.
Continue On Error — carry on instead of failing the flow. Defaults to false.

info

When Continue On Error is true the error is not raised at all, so a Catch node will not see it either.

Property	Field	Description
Client ID	`inClientID`	Client ID from Connect node (optional if credentials provided)
URL	`inURL`	Root URL to start crawling from

Property	Field	Description
Results	`outResults`	Array of crawled pages with URL and content

Property	Field	Description
Allow External	`optAllowExternal`	Include links to external domains
Credentials	`optCredentials`	Tavily API key (optional if Connect node is used)
Extract Depth	`optExtractDepth`	Advanced extracts tables and embedded content One of: `basic`, `advanced`.
Format	`optFormat`	Output format for extracted content One of: `markdown`, `text`.
Limit	`optLimit`	Total maximum number of pages to crawl
Max Breadth	`optMaxBreadth`	Maximum number of links to follow per level (1-500)
Max Depth	`optMaxDepth`	How many levels deep to crawl from the root URL (1-5)
Timeout (seconds)	`optTimeout`	Maximum time in seconds to wait

A Client ID from this package's Connect node, unless you set credentials directly on this node.
Tavily API Key — API key for Tavily AI search
- value — API Key (password, required). Your Tavily API key

Store these in a Vault and reference the vault item from the node, rather than typing the secret into the property.