Skip to main content

Generate Chat

Generates conversational AI responses using Google Vertex AI's PaLM 2 chat models (Chat Bison) with context and few-shot learning support.

Common Properties​

  • Name - The custom name of the node.
  • Color - The custom color of the node.
  • Delay Before (sec) - Waits in seconds before executing the node.
  • Delay After (sec) - Waits in seconds after executing node.
  • Continue On Error - Automation will continue regardless of any error. The default value is false.
info

If the ContinueOnError property is true, no error is caught when the project is executed, even if a Catch node is used.

Inputs​

  • Connection Id - Vertex AI client session identifier from Connect node (optional if credentials provided directly).
  • Credentials - Google Cloud service account credentials (optional if using Connection ID).
  • Project Id - Google Cloud Project ID (required if using direct credentials).
  • Author of The Message - Author role for the message (e.g., "user", "bot"). Default is "user".
  • Content of The Message - The actual content/text of the chat message to send (required).

Options​

Conversation Context​

  • Context - Context or system instructions for the chat model. Sets the behavior, tone, and constraints.
  • Examples Input - Example input for few-shot learning (must be paired with Examples Output).
  • Examples Output - Example output for few-shot learning (must be paired with Examples Input).

Generation Parameters​

  • Temperature - Controls randomness (0.0-1.0). Default is 0. Lower values are more deterministic.
  • Max Output Tokens - Maximum number of tokens to generate (1-2048). Default is 1024.
  • TopK - Top-K sampling parameter for token selection. Default is 40.
  • TopP - Top-P (nucleus) sampling parameter (0.0-1.0). Default is 0.95.
  • Stop Sequence - Array of sequences that stop generation when encountered. Optional.
  • Candidate Count - Number of response candidates to generate. Default is 1.

Model Configuration​

  • Model - Vertex AI chat model to use:
    • chat-bison@001 - Stable version for production (default)
    • chat-bison - Latest version with improvements
    • chat-bison-32k - Extended context window for longer conversations
    • Custom Model - Specify your own model name
  • Custom Model - Custom model name when "Custom Model" is selected.

Endpoint Configuration​

  • Locations - Google Cloud region for the Vertex AI endpoint. Default is "us-central1".
  • Publishers - Model publisher (typically "google"). Default is "google".

Output​

  • Response - Full API response object containing generated chat content.

Response structure:

{
  "predictions": [
    {
      "candidates": [
        {
          "author": "bot",
          "content": "Chat response here..."
        }
      ],
      "safetyAttributes": {
        "categories": [],
        "blocked": false,
        "scores": []
      },
      "citationMetadata": {
        "citations": []
      }
    }
  ]
}

How It Works​

The Generate Chat node creates conversational responses using Chat Bison models optimized for dialogue. When executed:

  1. Validates connection (either via Connection ID or direct credentials)
  2. Retrieves authentication token and project ID
  3. Validates message author and content are not empty
  4. Parses optional context and examples (if provided)
  5. Validates that Examples Input and Output are both provided or both empty
  6. Parses and validates generation parameters (temperature, tokens, etc.)
  7. Constructs request payload with message, context, examples, and parameters
  8. Sends POST request to Vertex AI chat predict endpoint
  9. Processes response and extracts chat content
  10. Returns complete response object with chat messages and metadata

Chat Bison models are specifically tuned for multi-turn conversations, maintaining context and providing natural, coherent responses.

Requirements​

  • Either:
    • Connection ID from Connect node, OR
    • Direct credentials + Project ID
  • Author of The Message (non-empty, typically "user")
  • Content of The Message (non-empty string)
  • Max Output Tokens (required, must be set)
  • Examples: Both Input and Output must be provided together, or both left empty
  • Vertex AI API enabled in Google Cloud project
  • IAM permissions: aiplatform.endpoints.predict

Error Handling​

Common errors and solutions:

ErrorCauseSolution
ErrInvalidArgEmpty message authorProvide valid author (e.g., "user")
ErrInvalidArgEmpty message contentProvide valid message text
ErrInvalidArgMax Output Tokens emptySet Max Output Tokens value
ErrInvalidArgExamples mismatchProvide both Input and Output, or leave both empty
ErrInvalidArgConnection ID or credentials missingUse Connect node or provide credentials
ErrNotFoundConnection not foundVerify Connection ID from Connect node
ErrStatusAPI error (quota, safety)Check Google Cloud Console for API status
Content blockedSafety filters triggeredAdjust message or review safety settings

Example Use Cases​

Customer Support Chatbot​

Context: "You are a helpful customer support agent for an e-commerce company. Be polite, professional, and solution-oriented."
Author: "user"
Content: "I haven't received my order yet, it's been 5 days"
Temperature: 0.3
Max Output Tokens: 200

Technical Documentation Assistant​

Context: "You are a technical documentation expert. Explain concepts clearly with examples."
Author: "user"
Content: "How do I configure OAuth2 authentication in my API?"
Examples Input: "What is REST API?"
Examples Output: "A REST API is an application programming interface that follows REST architectural principles..."
Temperature: 0.2
Max Output Tokens: 500

Educational Tutor​

Context: "You are a patient and encouraging math tutor for middle school students."
Author: "user"
Content: "Can you explain how to solve quadratic equations?"
Temperature: 0.4
Max Output Tokens: 400
Model: chat-bison-32k

Creative Writing Assistant​

Context: "You are a creative writing coach helping authors develop their stories."
Author: "user"
Content: "How can I make my protagonist more relatable?"
Temperature: 0.7
Max Output Tokens: 350
Candidate Count: 2

FAQ Automation​

Context: "You are an automated FAQ system. Provide concise, accurate answers based on company policy."
Examples Input: "What are your business hours?"
Examples Output: "We're open Monday-Friday, 9 AM to 6 PM EST. Customer support is available 24/7 via email."
Author: "user"
Content: "Do you offer international shipping?"
Temperature: 0.1
Max Output Tokens: 150

Tips​

  • Context is Key: Use Context to set role, tone, and behavior guidelines
  • Few-Shot Learning: Provide Examples to demonstrate desired response style
  • Author Roles: Use "user" for customer/user messages, "bot" for assistant responses
  • Temperature Settings:
    • 0.0-0.2: Factual, customer support, documentation
    • 0.3-0.5: Balanced conversations, general assistance
    • 0.6-0.8: Creative, brainstorming, entertainment
  • Model Selection:
    • chat-bison@001: Stable for production chatbots
    • chat-bison-32k: For conversations with long history
  • Token Management: Reserve tokens for multi-turn conversations
  • Conversation History: Manually maintain history by including previous messages in Context

Common Patterns​

Stateful Conversation​

Approach: Store conversation history
1. User sends message
2. Retrieve conversation history from database
3. Include history in Context field
4. Generate Chat response
5. Append user message + bot response to history
6. Store updated history
7. Display response to user

Intent Classification + Response​

Flow:
1. User message → Generate Chat (with intent classification context)
2. Parse response to detect intent
3. Route to appropriate handler based on intent
4. Generate final response with specific context

Multi-Language Support​

Context: "Respond in the same language as the user's question."
Use Case: Automatic language detection and response
Model: chat-bison-32k (better multilingual support)

Best Practices​

  • Context Design: Write clear, specific context instructions
  • Example Quality: Use high-quality, representative examples
  • Consistency: Maintain consistent author roles throughout conversations
  • Token Budget: Leave room for user input + context + response
  • Error Handling: Implement fallback responses for content blocking
  • State Management: Track conversation state outside the model
  • Testing: Test with diverse user inputs and edge cases
  • Safety: Monitor safety attributes in responses
  • Performance: Reuse connections for ongoing chat sessions
  • Versioning: Use chat-bison@001 for stable production deployments

Conversation Management​

Building Conversation History​

To maintain context across multiple messages, include previous messages in the Context field:

Context: "You are a helpful assistant.

Previous conversation:
User: What's the weather like?
Bot: I don't have access to real-time weather data.
User: Can you tell me a joke instead?"

Current message: "Make it about programming"

Managing Long Conversations​

  • Use chat-bison-32k for extended conversations
  • Summarize older messages to save tokens
  • Implement sliding window (keep last N messages)
  • Track total tokens across conversation

Response Validation​

Always check the response for:

  • Safety blocks
  • Complete responses (not truncated)
  • Appropriate tone and content
  • Citation metadata for factual claims