Key concepts
      Back to home
      1. Home
      2. 1: Introduction & Setup
      3. Key concepts

      Understanding ETL in Rayven.io

      ETL (Extract, Transform, Load) is a core capability in Rayven.io that helps you build real-time, automated workflows across CRMs, ERPs, databases, and support systems.

      Overview

      ETL (Extract, Transform, Load) is a core capability in Rayven.io that helps you build real-time, automated workflows across CRMs, ERPs, databases, and support systems. Whether you're synchronizing sales data, cleaning order logs, or enriching support tickets with AI, Rayven's ETL engine gives you a flexible, no-code way to build reliable data flows — all in real time.

      1. Extract: Bringing Data into Rayven

      ETL

      What It Does

      The Extract phase pulls data from diverse sources across your business systems.

      Supported Sources

      • CRMs (e.g., Salesforce, HubSpot)

      • ERPs (e.g., SAP, NetSuite)

      • Databases (MySQL, SQL Server, Snowflake)

      • Cloud Storage (e.g., AWS S3, Google Drive)

      • Web APIs (e.g., Zendesk, Stripe, Workday)

      How It Works

      • Use prebuilt connectors to configure real-time or scheduled ingestion

      • Set credentials, endpoints, and mapping rules

      • Ingest data as JSON, XML, CSV, or direct API streams

      Example: Ingest customer order updates from NetSuite using a scheduled API pull every 15 minutes.

      2. Transform: Applying Business Logic

      What It Does

      The Transform phase applies cleaning, enrichment, and calculation rules to ensure the data is usable and meaningful.

      Available Functions

      • Filtering: Remove duplicate or irrelevant entries

      • Aggregation: Summarize (e.g., total revenue per customer)

      • Calculations: Apply formulas (e.g., gross margin = revenue - COGS)

      • Custom Logic: Add JS or use rule builder nodes

      • Lookup Mapping: Enrich records with data from other tables (e.g., region codes, manager names)

      How It Works

      • Use Rayven’s Workflow Builder to chain logic visually

      • Configure nodes with formulas, thresholds, or group-by conditions

      • View real-time output previews to validate transformations

      Example: Clean a support log by removing empty fields, then calculate average resolution time by support rep.

      3. Load: Sending Data Where It’s Needed

      What It Does

      The Load phase delivers cleaned, processed data into downstream systems or UI components.

      Destinations

      • Dashboards: Display summaries in charts, KPIs, and tables

      • Rayven Tables: Store processed records for audits, analytics, or further automation

      • External APIs: Sync back into CRM, ERP, or reporting tools

      • Automated Alerts: Trigger emails, Slack messages, or system notifications

      How It Works

      • Use Output Nodes to configure your destination

      • Push full records or summaries

      • Trigger loads based on events, schedules, or workflow conditions

      Example: Push daily revenue summaries to a “RevenueSummary” table and auto-email finance managers a snapshot report.

      4. ETL + AI: Rayven's Intelligence Layer

      Real-Time Operation

      All ETL pipelines run continuously — perfect for time-sensitive operations like lead scoring, support prioritization, or compliance monitoring.

      ML Integration

      You can embed AI into your pipeline to predict or classify data:

      • Flag churn-risk customers

      • Predict next purchase date

      • Score leads from marketing systems

      Generative AI Integration

      Rayven lets you embed Generative AI into workflows to:

      • Create natural language summaries

      • Auto-generate reports or alerts

      • Enable conversational interfaces via chat widgets

      5. Example: ETL with CRM + Finance Integration

      1. Extract

      • Source: Customer interaction data from HubSpot and transaction data from Stripe

      • Method: API calls every 15 minutes

      2. Transform

      • Normalize timestamps and clean email fields

      • Aggregate total spend per customer

      • Detect duplicate records and consolidate

      3. Load

      • Store cleaned data in Rayven Tables

      • Push updated records into Salesforce via API

      • Display summary KPIs in the Sales dashboard

      4. Explain (Generative AI)

      • Generate summaries per customer:
        “Customer A had 12 new interactions and spent $4,200. No issues detected.”

      • Auto-send a daily digest to account managers

      • Enable chatbot queries:
        “Which enterprise clients spent more than $10k last week?”

      6. How Rayven ETL Differs from Traditional Tools

      Feature Traditional ETL Rayven ETL + GenAI
      Timing Batch Real-time
      AI Integration External + Manual Native ML and Generative AI
      Alerts Static thresholds Dynamic, human-readable notifications
      Interaction Code + pipeline configs Drag-and-drop + chatbot integration
       

      7. Frequently Asked Questions (FAQ)

      Q: Can I connect multiple business systems into one ETL pipeline?
      A: Yes. You can pull data from CRM, ERP, and databases into a single workflow.

      Q: Can Rayven clean and deduplicate records before syncing to Salesforce?
      A: Absolutely. Use transformation nodes to remove duplicates, fix email formats, and consolidate leads.

      Q: Can I automate end-of-day reports to my team?
      A: Yes. Use Generative AI to create summaries and Email or Slack nodes to send them automatically.

      Q: Do I need to write code to use Rayven ETL?
      A: No. You can build workflows visually, but JavaScript is available if advanced logic is needed.

      Q: Can I enrich support tickets with product or customer data during ETL?
      A: Yes. You can join or lookup additional fields from other tables or APIs during the transform phase.

      Q: Can Generative AI support different summaries per user group?
      A: Yes. Use filters or group IDs to generate role-specific summaries (e.g., finance, ops, marketing).