Skip to main content

Web Search

Searches the web using various search engines and returns formatted results with domain filtering.

Common Properties​

  • Name - The custom name of the node.
  • Color - The custom color of the node.
  • Delay Before (sec) - Waits in seconds before executing the node.
  • Delay After (sec) - Waits in seconds after executing node.
  • Continue On Error - Automation will continue regardless of any error. The default value is false.

Inputs​

  • Query - string - The search query (required, minimum 2 characters).
  • Allowed Domains - array - Only include search results from these domains (optional).
  • Blocked Domains - array - Never include search results from these domains (optional).

Options​

  • Max Results - int - Maximum number of search results to return (default: 10, max: 100).
  • Search Engine - Search engine to use: Mock (Demo), Google, Bing, or DuckDuckGo (default: Mock).
  • Region - string - Region code for localized results, e.g., us, uk, de (default: us).
  • Safe Search - bool - Enable safe search filtering (default: true).
  • Timeout - int - Request timeout in milliseconds (default: 10000).
  • API Key - API key for search engine (required for Google and Bing).
  • Use Robomotion Proxy - bool - Use Robomotion proxy for requests (default: false).

Outputs​

  • Results - array - Array of search results. Each result includes:
    • title - Page title
    • url - URL of the result
    • snippet - Description or excerpt
    • domain - Domain name
    • publishDate - Publication date (if available)
  • Result Count - int - Number of results returned.
  • Search Info - object - Information about the search:
    • query - The search query used
    • engine - Search engine used
    • region - Region setting
    • safeSearch - Safe search status
    • maxResults - Maximum results requested
    • actualResults - Actual number of results returned
    • filtered - Number of results filtered out by domain filters
    • executionTime - Search execution timestamp
    • allowedDomains - Applied domain allowlist
    • blockedDomains - Applied domain blocklist

How It Works​

The Web Search node performs web searches using various search engines. When executed, the node:

  1. Validates the search query (minimum 2 characters)
  2. Parses domain filter lists (allowed and blocked)
  3. Determines search parameters (max results, region, etc.)
  4. Performs search based on selected engine:
    • Mock: Returns demo results for testing
    • Google: Uses Google Custom Search JSON API
    • Bing: Uses Bing Web Search API v7
    • DuckDuckGo: Uses available DuckDuckGo API
  5. Applies domain filters:
    • Removes results from blocked domains
    • Keeps only results from allowed domains (if specified)
  6. Formats results with title, URL, snippet, domain, and date
  7. Returns filtered results and search metadata

Requirements​

  • Non-empty search query (minimum 2 characters)
  • For Google: Valid Google Custom Search API key
  • For Bing: Valid Bing Search API subscription key
  • For Google: Custom Search Engine ID configured
  • Network connectivity
  • For proxied requests: Robomotion proxy configuration

Error Handling​

The node will return specific errors in the following cases:

  • Missing query - "Query is required"
  • Query too short - "Query must be at least 2 characters long"
  • Search failed - "Search failed: {{error}}"

Usage Examples​

Query: robotic process automation
Search Engine: Mock

Search with Domain Filter​

Query: machine learning tutorials
Allowed Domains: ["github.com", "stackoverflow.com"]
Max Results: 20

Block Specific Domains​

Query: python programming
Blocked Domains: ["spam-site.com", "ads-site.com"]
Query: docker containers kubernetes
Search Engine: Google
API Key: [your API key]
Region: us
Max Results: 15
Query: artificial intelligence news
Search Engine: Bing
API Key: [your subscription key]
Safe Search: true

Search Engines​

Mock (Demo)​

  • No API key required
  • Returns sample results for testing
  • Useful for development and testing workflows
  • Results are not real search data

Google Custom Search​

  • Requires Google Cloud Platform account
  • Requires Custom Search JSON API enabled
  • Requires Custom Search Engine ID
  • Free tier: 100 queries/day
  • Paid tier: Higher limits available
  • High-quality results
  • Requires Microsoft Azure account
  • Requires Bing Search v7 resource
  • Free tier: Limited queries
  • Paid tier: Higher limits available
  • Good result quality

DuckDuckGo​

  • Privacy-focused search engine
  • Limited API availability
  • No official web search API
  • Implementation placeholder in current version

Domain Filtering​

Allowed Domains​

  • If specified: Only results from these domains are returned
  • If empty: All domains are allowed (except blocked)
  • Domain matching is substring-based
  • Examples: "github.com", "stackoverflow.com"

Blocked Domains​

  • Results from these domains are always filtered out
  • Applied before allowed domains filter
  • Domain matching is substring-based
  • Examples: "spam.com", "ads.example.com"

Filter Order​

  1. Check blocked domains first (remove matches)
  2. Check allowed domains next (keep only matches if list specified)
  3. Return filtered results

Usage Notes​

  • Query must be at least 2 characters long
  • Results are limited to Max Results setting (max 100)
  • Domain filters accept domain names without protocol
  • Domain matching is case-insensitive
  • Allowed Domains can be array or comma-separated string
  • Blocked Domains can be array or comma-separated string
  • Mock engine is useful for testing workflows
  • Real search engines require valid API keys
  • Search results include snippets (descriptions)
  • Publish dates are included when available from the search engine

Common Use Cases​

Research and Data Collection​

Gather information from the web on specific topics.

Competitive Analysis​

Monitor mentions of competitors or products.

Content Discovery​

Find relevant articles, tutorials, or resources.

Lead Generation​

Search for potential customers or partners.

Market Research​

Gather data about market trends and products.

SEO Monitoring​

Check search rankings for specific queries.

Best Practices​

  • Use specific queries for better results
  • Set appropriate max results based on needs
  • Filter domains to focus on quality sources
  • Block spam domains to improve result quality
  • Use region settings for localized results
  • Enable safe search for appropriate content
  • Cache results to avoid repeated searches
  • Rate limit API calls to stay within quotas
  • Monitor API usage to control costs
  • Handle empty results gracefully

API Setup​

Google Custom Search Setup​

  1. Create project in Google Cloud Console
  2. Enable Custom Search JSON API
  3. Create API credentials (API key)
  4. Create Custom Search Engine at google.com/cse
  5. Get Search Engine ID
  6. Configure API key in node

Bing Search Setup​

  1. Create Azure account
  2. Create Bing Search v7 resource
  3. Get subscription key from Azure portal
  4. Configure key in node
  5. Choose pricing tier based on volume

Cost Considerations​

Google Custom Search​

  • Free tier: 100 queries/day
  • Paid: $5 per 1000 queries (beyond free tier)

Bing Search​

  • Free tier: Limited queries (check current limits)
  • Paid: Based on pricing tier and volume

Mock Engine​

  • Completely free
  • No API costs
  • No query limits

Security Notes​

  • API keys should be stored securely
  • Use credential management for API keys
  • Avoid hardcoding API keys in workflows
  • Monitor API usage for unusual patterns
  • Use Safe Search for appropriate content
  • Respect search engine terms of service

Result Quality​

Tips for Better Results​

  • Use specific, descriptive queries
  • Include relevant keywords
  • Use quotation marks for exact phrases
  • Exclude unwanted terms with filters
  • Combine with domain filters for quality sources
  • Use region settings for local results

Comparison with Other Nodes​

  • Web Search vs Web Fetch: Web Search queries search engines; Web Fetch retrieves specific URLs
  • Web Search vs Grep: Web Search searches the web; Grep searches local files
  • Web Search vs Read: Web Search gets search results; Read reads local files

Future Enhancements​

Note: Current implementation includes placeholders for real search engines. Production implementation would include:

  • Full Google Custom Search API integration
  • Complete Bing Search API v7 integration
  • DuckDuckGo API integration (if available)
  • Additional search engines
  • Advanced query operators
  • Result pagination
  • Image and video search
  • News search specialization