List Bucket Objects

Retrieves a list of all objects (files) stored in a Google Cloud Storage bucket with their names and sizes.

Common Properties

• Name - The custom name of the node.
• Color - The custom color of the node.
• Delay Before (sec) - Waits in seconds before executing the node.
• Delay After (sec) - Waits in seconds after executing node.
• Continue On Error - Automation will continue regardless of any error. The default value is false.

info

If the ContinueOnError property is true, no error is caught when the project is executed, even if a Catch node is used.

Inputs

• GCS Client Id - The client identifier obtained from the Connect node. Optional if credentials are provided directly.
• Bucket Name - The name of the bucket to list objects from.

Output

• List - An array of objects containing file information. Each object includes:
  • Name - The full name/path of the object
  • Size - The size of the object in bytes

Stored in message context as list.

Options

• Credentials - Google Cloud service account credentials (optional). Use this instead of GCS Client Id for direct authentication without Connect node.

How It Works

The List Bucket Objects node retrieves all objects from a Google Cloud Storage bucket. When executed, the node:

1. Establishes connection using either GCS Client Id or provided credentials
2. Validates the bucket name
3. Queries Google Cloud Storage for all objects in the bucket
4. Iterates through all objects, collecting names and sizes
5. Returns an array with object details

Example

List All Objects

// After Connect node or with direct credentials
// Input:
//   Bucket Name: "my-data-bucket"

// Output in message.list:
// [
//   { Name: "reports/2024-01.csv", Size: 15234 },
//   { Name: "reports/2024-02.csv", Size: 16789 },
//   { Name: "config/settings.json", Size: 512 }
// ]

Process Each File

// Use Loop node to process each file
// After List Bucket Objects:

const files = message.list;
console.log(`Found ${files.length} files`);

// In Loop node:
files.forEach(file => {
  console.log(`File: ${file.Name}, Size: ${file.Size} bytes`);
  // Process each file (read, download, etc.)
});

Filter Large Files

// In JavaScript node after List Bucket Objects:

const largeFiles = message.list.filter(file => file.Size > 1000000); // > 1MB
console.log(`Large files: ${largeFiles.length}`);

message.largeFiles = largeFiles.map(f => f.Name);
// Then delete or process large files

Count Files by Type

// Analyze files in the bucket
const files = message.list;

const fileTypes = {};
files.forEach(file => {
  const ext = file.Name.split('.').pop();
  fileTypes[ext] = (fileTypes[ext] || 0) + 1;
});

console.log('File types:', fileTypes);
// Output: { csv: 15, json: 3, txt: 7 }

Calculate Total Storage

// Calculate total storage used
const files = message.list;

const totalBytes = files.reduce((sum, file) => sum + file.Size, 0);
const totalMB = (totalBytes / 1024 / 1024).toFixed(2);

console.log(`Total storage: ${totalMB} MB`);
console.log(`Number of files: ${files.length}`);

Requirements

• Either a valid GCS Client Id from Connect node OR credentials provided directly
• Valid bucket name that exists in Google Cloud Storage
• Appropriate IAM permissions:
  • storage.objects.list permission
  • roles/storage.objectViewer or higher

Error Handling

The node will return specific errors in the following cases:

Error Code	Description
ErrInvalidArg	Bucket Name is empty or invalid

Common error scenarios:

• Empty or invalid GCS Client Id without credentials
• Empty or invalid Bucket Name
• Bucket doesn't exist
• Insufficient permissions to list objects
• Google Cloud Storage service errors
• Network connectivity issues

Usage Notes

• Returns an array with Name and Size for each object
• Objects in folder-like paths are listed with full paths (e.g., "folder/file.txt")
• All objects in the bucket are returned, regardless of storage class
• Large buckets (100,000+ objects) may take time to list
• The operation is read-only and doesn't modify objects
• Empty buckets return an empty array []
• The list is not sorted by default - sort in JavaScript if needed
• Size is always in bytes - convert to KB/MB/GB as needed

Tips for Effective Use

• Use with Loop nodes to process multiple files
• Filter results based on file names or sizes
• Implement pagination for very large buckets
• Cache results to avoid repeated API calls
• Use for bucket inventory and auditing
• Combine with other nodes for batch operations
• Sort results by name or size as needed
• Use prefix patterns in file names for easier filtering
• Monitor large buckets for performance

Common Errors and Solutions

Error: "Bucket Name cannot be empty"

• Solution: Ensure bucket name is properly set
• Check variable bindings

Error: Insufficient permissions

• Solution: Verify service account has storage.objects.list permission
• Check bucket IAM policies

Empty list returned but bucket has files

• Solution: Verify bucket name is correct (case-sensitive)
• Check if using the correct GCS project
• Verify permissions to view objects

Timeout or slow response

• Solution: Bucket may have many objects (100,000+)
• Consider implementing prefix-based filtering
• Use pagination strategies

Use Cases

Inventory Management

• Create inventory of all files in a bucket
• Audit storage usage and file counts
• Generate reports on bucket contents

Batch Processing

• List all files before batch operations
• Process all CSV files in a bucket
• Download multiple files in sequence

Storage Analytics

• Calculate total storage used
• Identify large files consuming space
• Analyze file types and distributions

Data Validation

• Verify expected files are present
• Check for missing or extra files
• Validate file naming conventions

Automated Cleanup

• Find old files for deletion
• Identify temporary files to remove
• Locate duplicate files

Monitoring and Alerts

• Monitor bucket file counts
• Alert on unexpected file counts
• Track storage growth over time

Advanced Examples

Find Files by Date Pattern

// Find files matching date pattern
const files = message.list;
const datePattern = /2024-01/; // Files from January 2024

const januaryFiles = files.filter(file =>
  datePattern.test(file.Name)
);

console.log(`January files: ${januaryFiles.length}`);
message.filesToProcess = januaryFiles;

Sort by Size

// Sort files by size (largest first)
const files = message.list;

const sortedFiles = files.sort((a, b) => b.Size - a.Size);

console.log('Largest files:');
sortedFiles.slice(0, 10).forEach(file => {
  const sizeMB = (file.Size / 1024 / 1024).toFixed(2);
  console.log(`${file.Name}: ${sizeMB} MB`);
});

Group by Folder

// Group files by folder
const files = message.list;

const folders = {};
files.forEach(file => {
  const folder = file.Name.includes('/')
    ? file.Name.split('/')[0]
    : 'root';

  if (!folders[folder]) folders[folder] = [];
  folders[folder].push(file);
});

console.log('Files per folder:',
  Object.keys(folders).map(f => `${f}: ${folders[f].length}`)
);