Alex Merced's Lakehouse Blog

Context Management Strategies for VS Code with LLM Plugins: A Complete Guide to Building Your Own AI-Powered IDE

Alex Merced — Sun, 08 Mar 2026 01:00:00 GMT

Visual Studio Code is the most widely used code editor in the world, and its extensibility means you can integrate AI capabilities through a growing ecosystem of LLM plugins. Unlike purpose-built AI editors (Cursor, Windsurf, Zed), VS Code gives you the freedom to choose and combine AI extensions, configure them to your preferences, and even switch between providers without changing editors. The tradeoff is that context management is not as seamlessly integrated as in dedicated AI editors. It requires more deliberate configuration.

This guide covers context management strategies for the most popular VS Code AI extensions: GitHub Copilot, Continue, Cline (formerly Claude Dev), Aider, and others. It explains what context management capabilities each offers and how to configure them for maximum effectiveness.

The VS Code AI Extension Landscape

VS Code's AI extension ecosystem falls into several categories:

Category	Extensions	Approach
Inline completion	GitHub Copilot, CodeiumChat, Supermaven	Suggest code as you type
Chat panel	Copilot Chat, Continue, Cody	Conversational AI in a sidebar
Agentic coding	Cline, Aider, Roo Code	Autonomous agents that read/write files
Specialized	Mintlify, Tabnine	Documentation, enterprise-focused

Each category manages context differently. Inline completion plugins use the current file and nearby tabs. Chat panel plugins use conversation history and file references. Agentic plugins have the broadest context, reading the codebase, running commands, and making multi-file changes.

GitHub Copilot: Context Management

GitHub Copilot is the most widely used AI coding assistant. Its context management has evolved significantly with the introduction of Copilot Chat and Agent Mode.

Inline Completions

Copilot's inline suggestions use:

The current file content (especially the lines around your cursor)
Open tabs in the editor (nearby files provide additional context)
File names and directory structure (for naming conventions)
Comment and docstring context (comments above your cursor guide suggestions)

Tip: Keep related files open in tabs. Copilot considers open files as context, so having related source files, type definitions, and tests open improves suggestion quality.

Copilot Chat

Copilot Chat operates in the sidebar with conversation-based interaction:

Use #file to reference specific files
Use #editor to reference the active editor content
Use #selection to reference selected code
Use #codebase to search the workspace
Use @workspace to ask questions about the entire project

Copilot Agent Mode

Agent Mode (introduced in 2025) makes Copilot an autonomous agent that can:

Plan multi-step changes
Read and write files across the project
Run terminal commands
Make and verify changes iteratively

Agent Mode uses the broadest context of any Copilot feature: it can explore the codebase, read package.json, check test results, and understand project structure before making changes.

Custom Instructions for Copilot

Create a .github/copilot-instructions.md file in your project root:

# Copilot Instructions

## Code Style
- Use TypeScript strict mode
- Prefer functional components with hooks
- Use named exports, not default exports
- Follow the Airbnb ESLint configuration

## Testing
- Write tests using Vitest
- Use React Testing Library for component tests
- Mock API calls with MSW

## Architecture
- Components go in src/components/
- API clients go in src/api/
- Shared types go in src/types/

These instructions are loaded by Copilot for every interaction within the project, functioning like .cursor/rules/ in Cursor.

Continue: Open-Source AI Extension

Continue is an open-source VS Code extension that supports multiple LLM providers and offers extensive context management features.

Provider Configuration

Continue supports:

OpenAI, Anthropic, Google models via API keys
Ollama for local models
Any OpenAI-compatible endpoint

Context Providers

Continue's "@-mention" context system includes:

Context Provider	Function
`@file`	Include a specific file
`@code`	Include code blocks from the codebase
`@docs`	Search indexed documentation
`@codebase`	Semantic search across the project
`@terminal`	Include recent terminal output
`@diff`	Include current Git diff
`@repo`	Include repository metadata
`@folder`	Include folder structure

.continuerc.json Configuration

{
  "models": [
    {
      "title": "Claude Sonnet",
      "provider": "anthropic",
      "model": "claude-sonnet-4-20250514",
      "apiKey": "your-key"
    },
    {
      "title": "Local Llama",
      "provider": "ollama",
      "model": "llama3.1:70b"
    }
  ],
  "customCommands": [
    {
      "name": "review",
      "prompt": "Review this code for security issues, performance problems, and style violations."
    }
  ],
  "docs": [
    {
      "title": "React Docs",
      "startUrl": "https://react.dev/reference"
    }
  ]
}

Why Continue Stands Out for Context

Continue's open-source nature means you can inspect exactly how context is assembled. Its support for custom context providers extends beyond built-in options, allowing teams to create project-specific context sources.

Cline (formerly Claude Dev): Agentic Coding Agent

Cline is a VS Code extension that turns Claude into an autonomous coding agent within the editor.

Context Capabilities

Cline has one of the broadest context scopes among VS Code extensions:

Reads and writes files across the entire project
Runs terminal commands
Browses the web (for documentation lookup)
Takes screenshots of running applications
Manages its own task history

Project Instructions

Create a .clinerules file in your project root:

# Project: SaaS Application

## Stack
- Python 3.12 with FastAPI
- PostgreSQL with SQLAlchemy
- Redis for caching
- React frontend with TypeScript

## Build Commands
- Backend: `uvicorn app.main:app --reload`
- Frontend: `npm run dev`
- Tests: `pytest -v`

## Conventions
- All API responses use the ResponseModel pattern
- Database sessions are managed by dependency injection
- Frontend state uses React Query for server state

Custom MCP Servers

Cline supports MCP servers configured through its settings panel, enabling connections to databases, APIs, and other external tools directly within the VS Code environment.

Context Window Management

Cline tracks context window usage and can summarize previous conversation history when the window fills up. This automatic context management prevents the common problem of long sessions degrading quality.

Aider: Git-Aware AI Pair Programmer

Aider integrates with VS Code as a terminal-based tool that focuses on Git-aware code modifications.

Context Management in Aider

Aider uses a unique context model:

Chat files: Files actively being discussed and modified
Watch files: Files included as read-only context
Repository map: An overview of the entire repository structure that fits in context

Commands for Context Control

/add src/auth/middleware.ts    # Add to chat context (can be edited)
/read docs/architecture.md     # Add as read-only context
/drop src/auth/middleware.ts   # Remove from context
/map                           # Show the repository map

The Repository Map

Aider's repository map is a compressed representation of your entire codebase (file names, function signatures, class definitions) that fits within the context window. This gives the AI a bird's-eye view of the project without consuming the entire context budget.

Thinking About Context Levels Across Extensions

Minimal Context (Quick Completions)

For inline code completions, Copilot and Supermaven work well with minimal setup. Keep related files open in tabs and let the extension use the editor context.

Moderate Context (Feature Development)

Use a chat extension (Copilot Chat, Continue) with explicit file references. The @-mention system lets you include exactly the files relevant to the current task.

Comprehensive Context (Major Refactoring)

Use an agentic extension (Cline, Copilot Agent Mode) that can explore the codebase, run tests, and make changes across multiple files. Configure project instructions (.clinerules, copilot-instructions.md, .continuerc.json) to ensure the agent follows your conventions.

External Documents: PDFs vs. Markdown

Markdown Is Universal

All VS Code AI extensions work natively with Markdown. Project instructions, coding standards, and architecture documents should be Markdown files in your repository.

PDFs

Most VS Code extensions do not parse PDFs directly. If you have reference material in PDF form, extract relevant sections into Markdown files. Some extensions (like Cline with web browsing) can fetch online documentation, reducing the need for local PDF conversion.

Documentation Indexing

Continue and Copilot Chat support documentation indexing through @docs. Add your framework documentation URLs to the extension configuration so the AI can reference current documentation during conversations.

MCP Server Support

MCP support varies by extension:

Extension	MCP Support	Configuration
Cline	Yes	Settings panel
Continue	Yes	config.json
Copilot	Limited	Through GitHub integration
Aider	No	Direct terminal commands instead

For extensions that support MCP, the configuration follows the standard pattern: specify the server command, arguments, and environment variables. MCP tools become available within the extension's chat or agent interface.

settings.json: Centralizing AI Configuration

VS Code's settings.json is where many AI extensions read their configuration. Here are common settings patterns:

Per-Workspace Settings

Create a .vscode/settings.json file in your project to configure AI extensions per-project:

{
  "github.copilot.enable": {
    "markdown": true,
    "plaintext": false
  },
  "continue.enableTabAutocomplete": false,
  "cline.customInstructions": "Follow the conventions in INSTRUCTIONS.md"
}

Per-workspace settings override user-level settings, allowing you to tailor AI behavior to each project's needs.

Workspace Trust and Security

VS Code's Workspace Trust feature is important when using AI extensions. In untrusted workspaces, some extensions may limit their capabilities (for example, restricting file access or command execution). This is a security feature: it prevents untrusted code from being automatically processed by AI tools that have file system access.

For your own projects, trust the workspace to enable full AI capabilities. For third-party codebases, consider the implications before trusting.

When to Use VS Code with Plugins vs. Dedicated AI Editors

Choose VS Code with plugins when:

You already use VS Code and want to add AI incrementally
You want to mix and match extensions from different providers
You have existing VS Code extensions and workflows you cannot replicate elsewhere
You need the specific capabilities of an extension that only exists for VS Code (like Cline)
Your team uses different AI providers and needs a common editor

Choose Cursor or Windsurf when:

You want the most seamlessly integrated AI experience
You prefer automatic codebase indexing over manual context management
You are starting fresh and do not have an existing VS Code extension stack
You want features like .cursor/rules/ or Cascade flows that are deeply integrated

Choose a terminal agent (Claude Code, Gemini CLI) when:

Your workflow is terminal-centric
You need direct shell command execution as your primary interaction
You prefer a focused, distraction-free coding experience

Advanced Patterns

The Multi-Extension Stack

Use multiple extensions simultaneously for different purposes:

Copilot for inline completions (fast, low-friction)
Continue for chat with @codebase search (exploratory questions)
Cline for agentic tasks (multi-file changes, complex features)

Each extension handles a different level of context and interaction.

The Consistent Instructions Pattern

Maintain a single INSTRUCTIONS.md file in your project root and reference it from each extension's configuration:

.github/copilot-instructions.md imports or mirrors INSTRUCTIONS.md
.continuerc.json references INSTRUCTIONS.md
.clinerules mirrors the same conventions

This ensures consistent behavior regardless of which extension handles the task.

The Provider Rotation Pattern

Use different providers for different extensions:

Copilot: GitHub's infrastructure (fast, always available)
Continue: Anthropic API (strong at code analysis)
Cline: Local Ollama model (privacy for sensitive code)

This gives you the benefits of multiple providers within a single editor.

Common Mistakes

Using too many AI extensions simultaneously. Running five AI extensions creates conflicts, performance overhead, and conflicting suggestions. Pick a primary stack and disable the rest.
Not configuring project instructions. Every AI extension supports some form of project-level instructions. Without them, the AI relies on generic conventions.
Ignoring @codebase search. Both Copilot Chat and Continue offer codebase search. Using it produces more relevant responses than manually specifying files.
Not keeping related tabs open. Inline completion quality improves when related files are open in the editor. Keep type definitions, tests, and related source files in your tab bar.
Choosing the wrong extension for the task. Inline completions for quick code, chat for questions, agent mode for complex changes. Match the tool to the task.
Skipping documentation indexing. If you are working with a framework, index its documentation so the AI references current, accurate information rather than potentially outdated training data.

Go Deeper

To learn more about AI-assisted development and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for T3 Chat: A Complete Guide to the Unified Multi-Model AI Interface

Alex Merced — Sun, 08 Mar 2026 00:00:00 GMT

T3 Chat is a modern web-based AI chat interface that gives you access to multiple AI models through a single unified platform. Its primary value proposition is model flexibility: instead of being locked into one provider, you can switch between Claude, GPT, Gemini, Llama, and other models within the same interface. This makes T3 Chat unique from a context management perspective because the same context strategies must work across fundamentally different model families with different capabilities, context window sizes, and strengths.

This guide covers how to manage context effectively in T3 Chat to get the most from its multi-model architecture, from conversation organization to system prompts and file handling.

How T3 Chat Manages Context

T3 Chat builds its context from several sources:

System prompts - persistent instructions that shape every response
Model selection - the underlying model determines context window and capabilities
Conversation history - the message thread within the current chat
File attachments - documents and images uploaded to conversations
Personas - saved configurations combining system prompts with preferred models
Folders and organization - conversation grouping for project-based workflows

The context management challenge unique to T3 Chat is that different models interpret your context differently. A system prompt that works well with Claude may need adjustment for GPT or Gemini. Understanding these differences helps you write model-portable context.

System Prompts: The Foundation

T3 Chat supports custom system prompts that you set per-conversation or through Personas.

Writing Effective System Prompts

You are a senior software architect with expertise in distributed systems.

## Response Style
- Be technical and precise
- Include code examples when relevant
- Use bullet points for lists of recommendations
- Explain tradeoffs, do not just give the "right" answer

## Constraints
- Assume the reader has 5+ years of programming experience
- Do not explain basic concepts unless asked
- When discussing frameworks, focus on architectural implications, not syntax tutorials

## Output Format
- Use headers to organize long responses
- Include a "Key Takeaway" section at the end of detailed analyses
- Format code blocks with language annotations

Model-Portable System Prompts

Because T3 Chat supports multiple models, write system prompts that work across model families:

Be explicit about format expectations. Different models interpret vague formatting instructions differently.
Avoid model-specific references. Do not write "As Claude, you should..." or "Using your GPT capabilities..."
Focus on behavior and output. Describe what you want the model to do, not how you think it should reason internally.
Test across models. Send the same prompt to Claude, GPT, and Gemini within T3 Chat to verify consistent behavior.

Personas: Reusable Context Configurations

Personas combine a system prompt with a preferred model selection into a reusable configuration. Think of them as "modes" you can switch between.

Creating Effective Personas

Persona	System Prompt Focus	Model Choice
Code Reviewer	Security, performance, style guide checks	Claude Sonnet (strong at code analysis)
Technical Writer	Documentation standards, audience awareness	GPT-4o (strong at prose)
Research Analyst	Citation requirements, source evaluation	Gemini Pro (strong at retrieval and synthesis)
Creative Brainstormer	Divergent thinking, idea generation	Claude Opus or GPT-4o (creative capabilities)

When to Create Personas

Create a Persona when you find yourself:

Repeating the same system prompt across conversations
Switching to the same model for a specific type of task
Wanting to standardize how the AI handles a particular workflow

Personas save time and ensure consistency. Instead of re-configuring the system prompt and model for each new conversation, select the appropriate Persona and start working.

Model Selection as Context Management

Choosing the right model in T3 Chat is itself a context management decision because different models have different context window sizes and capabilities.

Context Window Comparison

Model	Approximate Context Window	Strengths
Claude Sonnet	200K tokens	Long context, code analysis, nuanced reasoning
Claude Opus	200K tokens	Complex analysis, creative writing
GPT-4o	128K tokens	Broad capabilities, strong at prose and instruction following
GPT-o3	200K tokens	Deep reasoning, complex problem solving
Gemini Pro	1M+ tokens	Massive context, document analysis
Llama 3.1 (70B)	128K tokens	Open source, privacy-friendly

Model Selection Strategy

For T3 Chat users, the model selection strategy directly affects context management:

Long documents or many files: Choose Gemini Pro (massive context window) or Claude (200K)
Quick questions: Choose a fast model (GPT-4o-mini, Claude Haiku) for responsiveness
Privacy-sensitive content: Choose Llama through a local endpoint
Complex analysis: Choose Claude Opus or GPT-o3 for deep reasoning

Being deliberate about model selection means your context is used more effectively by a model suited to the task.

Conversation Organization

T3 Chat provides tools for organizing your conversations into a structured workspace.

Folders

Group conversations by project, topic, or workflow. This is not just for tidiness; organized conversations make it easier to find and resume context:

/projects/web-app/ might contain conversations about frontend, backend, and deployment
/research/market-analysis/ might contain conversations about different market segments
/writing/blog-series/ might contain conversations for each blog post

Pinned Conversations

Pin important conversations for quick access. Pin your most frequently referenced threads so you can revisit them without searching.

Naming Conventions

Name conversations descriptively:

"Auth module refactoring plan" is searchable and findable
"New chat 47" is neither

Good naming is a form of context management because it makes your accumulated knowledge retrievable.

File Attachments

T3 Chat supports file uploads for providing document-level context within conversations.

Supported File Types

Documents: PDFs, Markdown, plain text
Images: Screenshots, diagrams, mockups
Spreadsheets: CSV, Excel files for data analysis
Code files: Source code in any language

Best Practices for File Attachments

Upload only the files relevant to the current question. Uploading your entire project creates noise.
For long documents, tell the model which sections to focus on: "This is our API specification. Focus on the authentication endpoints in Section 3."
For images, provide a text description of what the model should look for: "This is a screenshot of our dashboard. The chart in the upper-right shows incorrect data."

External Documents: PDFs vs. Markdown

PDFs

T3 Chat can process PDFs uploaded as attachments. PDFs work well for:

Formal documents (research papers, specifications, contracts)
Published content with fixed formatting
Multi-page documents with embedded images and tables

Markdown

For context you author specifically for the AI (system prompts, reference documents, instructions), Markdown is cleaner:

Models parse Markdown more reliably than extracted PDF text
Markdown is easier to version and update
The structure (headings, lists, code blocks) is explicit, not inferred

The Practical Rule

If the document exists as a PDF and you cannot easily convert it, upload the PDF. If you are writing the document for the purpose of giving it to the AI, write it in Markdown.

MCP Server Support

T3 Chat supports MCP (Model Context Protocol) server connections, allowing the platform to integrate with external data sources and tools. This extends T3 Chat's capabilities beyond conversation and file uploads by enabling connections to services like Google Drive, Slack, GitHub, databases, and custom APIs.

How MCP Works in T3 Chat

MCP servers provide T3 Chat with access to external resources and tools. When configured, the AI can query external data sources, retrieve real-time information, and perform actions through connected services. This makes T3 Chat more than just a chatbot: it becomes an interface for interacting with your broader tool ecosystem.

When MCP Adds Value

MCP is most useful in T3 Chat when your conversations need live data access:

Querying a database while discussing architecture decisions
Accessing project management data during planning conversations
Retrieving documentation from connected services
Interacting with APIs through a conversational interface

For conversations that rely purely on the AI's training data or uploaded files, MCP is unnecessary. It adds the most value when you need real-time connections to external systems during your conversations.

Thinking About Context Levels in T3 Chat

Quick Questions (Minimal Context)

For factual or conceptual questions, just ask. No special setup needed:

"What is the difference between horizontal and vertical scaling in database architecture?"

The model's training data is sufficient context, and no files or custom prompts are required.

Working Sessions (Moderate Context)

For sustained work on a topic, create a conversation with an appropriate Persona and provide reference files:

"I am building a REST API for a healthcare application. Here is the data model [attach file]. Help me design the endpoints following HIPAA compliance patterns."

Complex Projects (Comprehensive Context)

For multi-day projects, create a folder of organized conversations, use Personas for different phases of work, and bridge context between conversations using explicit summaries.

Model-Specific Context Tuning

Each model family responds slightly differently to the same context. Here are practical tips for tuning:

Claude in T3 Chat

Responds well to role-based system prompts ("You are a...")
Handles very long contexts gracefully
Benefits from explicit format instructions

GPT Models in T3 Chat

Follows formatting instructions precisely
Works well with example-based prompts ("Here is an example of what I want: ...")
Benefits from numbered constraints

Gemini in T3 Chat

Excels with document analysis tasks
Handles massive context windows (1M+ tokens)
Benefits from clear section headers in system prompts

When to Use T3 Chat vs. Other Tools

Use T3 Chat when:

You want to compare responses across different models
You need flexible model selection without multiple subscriptions
Your task is conversational (research, analysis, writing, brainstorming)
You want Personas for reusable configurations

Use a coding IDE (Cursor, Windsurf, Zed) when:

Your task involves editing code files directly
You need workspace indexing and @codebase search
You want agent mode to make cross-file changes

Use a terminal agent (Claude Code, Gemini CLI) when:

You need direct terminal access and command execution
Your task involves running tests, builds, or deployments

Advanced Patterns

The Model Comparison Pattern

Use T3 Chat's multi-model support to compare responses:

Ask the same question to Claude, GPT, and Gemini
Compare the responses for accuracy, depth, and style
Use the best response as a starting point and refine it

This is especially useful for high-stakes content where you want multiple perspectives before finalizing.

The Persona Pipeline Pattern

Chain Personas for multi-step work:

Research Persona (Gemini): Gather information and sources
Analysis Persona (Claude): Analyze the research and identify key themes
Writing Persona (GPT): Draft the final output based on the analysis

Each step uses a model optimized for that type of work, with context transferred manually between conversations.

The Context Bridging Pattern

When switching between models in the same conversation, bridge the context explicitly:

"Here is a summary of what we discussed so far: [summary]. I am switching to a different model. Please continue from this point."

This helps the new model pick up the thread without losing continuity.

Common Mistakes

Not using Personas for repeatable work. If you are configuring the same system prompt and model combination repeatedly, create a Persona.
Ignoring model differences. Claude, GPT, and Gemini respond differently to the same prompt. If results are not meeting expectations, try a different model before rewriting the prompt.
Uploading too many files. Each file consumes context window space. Be selective and upload only what is relevant to the current question.
Not organizing conversations. Without folders and descriptive names, your accumulated research and context becomes unfindable as conversations accumulate.
Using the same model for everything. T3 Chat's strength is model flexibility. Use Gemini for massive documents, Claude for code analysis, and GPT for prose generation.
Writing model-specific system prompts. If your system prompt only works with one model, it is too model-specific. Write instructions that describe behavior and output, not internal reasoning.

Go Deeper

To learn more about working effectively with AI interfaces and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Zed: A Complete Guide to the High-Performance AI Code Editor

Alex Merced — Sat, 07 Mar 2026 23:00:00 GMT

Zed is a high-performance code editor built in Rust that prioritizes speed, simplicity, and real-time collaboration. Its AI integration is designed to be fast and unobtrusive, with context management built around an assistant panel, inline transformations, slash commands, and a flexible provider system that supports multiple AI services. What sets Zed apart from other AI editors is its focus on performance (everything runs natively, not in Electron) and its built-in multiplayer editing that extends to AI interactions.

This guide covers how to manage context effectively in Zed's AI features to get the most from its lightweight but capable AI integration.

How Zed Manages Context

Zed builds AI context from several sources:

Assistant panel - a dedicated panel for multi-turn conversations with persistent context threads
Inline transformations - context-aware edits triggered in the editor
Slash commands - special commands that inject structured context into prompts
Active buffers - files currently open in the editor
Project structure - the workspace file tree
Custom prompts library - saved, reusable prompt templates
Language server data - type information and diagnostics from LSPs
MCP servers - external tool connections (supported in recent versions)

Zed takes a minimalist approach to context management: rather than automatically indexing your entire codebase (like Cursor or Windsurf), it gives you explicit control over what goes into context through slash commands and file references.

The Assistant Panel: Structured Conversations

Zed's Assistant Panel is the primary interface for AI interactions that require context beyond the current file. It operates as a structured conversation where you build context explicitly.

How the Panel Works

The panel displays a conversation thread where each message can include code blocks, file references, and slash command outputs. You compose messages, include context, and receive AI responses in a single, reviewable flow.

Persistent Context Threads

Each conversation in the panel is a persistent thread. You can name threads, save them, and return to them later. This means you can maintain ongoing conversations about specific features or architectural decisions without losing context between sessions.

Including Code from Open Buffers

You can drag files or code selections into the assistant panel to include them as context. This explicit inclusion model means you always know exactly what context the AI is working with, unlike tools that silently assemble context behind the scenes.

Zed's Explicit Context Philosophy

Zed's approach to context management is fundamentally different from editors like Cursor or Windsurf that automatically index and retrieve context. In Zed, you explicitly choose what context to provide through slash commands and file inclusions. This has important tradeoffs:

Advantages of explicit context:

You always know what the AI is working with
No surprises from irrelevant code being included
Works well with smaller model context windows (no wasted tokens)
Context is reproducible: the same slash commands always produce the same context

Tradeoffs:

Requires more manual effort to set up context
You need to know which files are relevant before asking
The AI cannot discover related code on its own (unlike @codebase in other editors)

Understanding this philosophy helps you use Zed's AI features effectively: invest time in selecting the right context rather than expecting the editor to figure it out for you.

Real-Time Collaboration and AI

Zed's built-in multiplayer editing extends to AI interactions. When collaborating in a shared workspace:

Multiple developers can contribute to the same assistant panel conversation
One developer can set up the context while another frames the question
AI suggestions can be reviewed and discussed collaboratively in real time
The AI's output is visible to all participants simultaneously

This makes Zed uniquely suited for pair programming and team code review workflows that incorporate AI assistance.

Slash Commands: Explicit Context Injection

Slash commands are Zed's primary mechanism for injecting specific types of context into AI conversations.

Available Slash Commands

Command	Function
`/file [path]`	Include a specific file's content
`/tab`	Include all currently open tabs
`/diagnostics`	Include current LSP errors and warnings
`/search [query]`	Search the project and include results
`/prompt [name]`	Load a saved prompt template
`/now`	Include the current date and time
`/fetch [url]`	Fetch and include content from a URL

Using Slash Commands Effectively

The power of slash commands is precision. Instead of sending your entire codebase as context, you choose exactly which files and information are relevant:

/file src/auth/middleware.ts
/file src/auth/types.ts
/diagnostics

I need to fix the TypeScript errors in the auth middleware.
The types file defines the expected interfaces.

This focused approach produces better results than sending the AI a vague question against a massive context window. Each piece of context is intentional and relevant.

/diagnostics for Error-Driven Context

The /diagnostics command is particularly powerful because it pulls language server errors and warnings directly into the AI conversation. Instead of manually copying error messages, one command gives the AI structured diagnostic information.

/fetch for External Documentation

The /fetch command retrieves content from URLs, making it easy to include external documentation, API specifications, or reference material without manual copying:

/fetch https://docs.myframework.com/api/routing

How do I implement nested routing using this framework's API?

Custom Prompts Library

Zed maintains a library of saved prompts that you can reuse across conversations and projects.

Creating Custom Prompts

Navigate to the prompts library and create templates for common tasks:

# Code Review Template

Review the provided code for:
1. Security vulnerabilities (injection, XSS, CSRF)
2. Performance issues (N+1 queries, unnecessary allocations)
3. Error handling completeness
4. Type safety issues
5. Missing edge cases

For each issue found:
- Describe the problem
- Explain the risk
- Provide a fix

Using Prompts

Load a saved prompt with the /prompt slash command:

/prompt code-review
/file src/api/users.ts

This combines your predefined review criteria with the specific file, creating a structured, repeatable workflow.

When to Create Prompts

Create prompts for tasks you perform regularly:

Code reviews with consistent criteria
Documentation generation in a specific format
Refactoring with specific patterns (extract function, apply interface)
Test generation following your testing conventions

AI Provider Configuration

Zed supports multiple AI providers, giving you flexibility in model selection:

Supported Providers

Provider	Configuration	Notes
Anthropic	API key in settings	Claude models
OpenAI	API key in settings	GPT models
Ollama	Local endpoint	Private, local models
Google	API key in settings	Gemini models
OpenRouter	API key in settings	Multi-provider routing
Custom	Any OpenAI-compatible endpoint	Self-hosted models

Context Window Implications

Different providers offer different context window sizes. With Zed's explicit context management (where you choose what to include via slash commands), you have good visibility into how much context you are using. If you are working with a smaller model through Ollama, be more selective with your slash commands. With a large cloud model, you can include more files.

Configuring in settings.json

{
  "language_model": {
    "provider": "anthropic",
    "model": "claude-sonnet-4-20250514"
  }
}

Inline Transformations

For quick edits that do not require a full conversation, Zed's inline transformation feature lets you select code and apply AI-powered changes directly in the editor.

How It Works

Select code in the editor
Trigger the inline transformation (keyboard shortcut)
Type your instruction ("Add error handling" or "Convert to async/await")
Zed applies the change inline

Context for Inline Transformations

Inline transformations use a focused context: the current file, the selection, and your instruction. They do not load your custom prompts or conversation history. This makes them fast and appropriate for small, self-contained changes.

MCP Server Support

Recent versions of Zed support MCP for connecting to external tools. The implementation follows the standard MCP pattern: configure servers in settings, and their tools become available within the assistant panel.

Configuration

{
  "context_servers": {
    "postgres": {
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-server-postgres"]
    }
  }
}

When to Use MCP in Zed

MCP is most useful when the assistant needs live data (database schemas, API responses, running service status) that cannot be obtained from static files. For code-only tasks, the slash commands and file references are sufficient.

External Documents: PDFs vs. Markdown

Markdown for Everything You Control

Prompts, reference documents, and coding standards should be Markdown. Zed's prompt library and slash commands work natively with text-based formats.

PDFs

Zed does not have built-in PDF parsing. For reference material in PDF form, extract relevant sections into Markdown files in your project and reference them with /file. Alternatively, use /fetch if the content is available online.

Thinking About Context Levels in Zed

Minimal Context (Inline Edits)

Select code, trigger inline transformation, describe the change. The current file and selection provide sufficient context for small changes.

Moderate Context (Feature Work)

Use the assistant panel with targeted slash commands: /file for relevant files, /diagnostics for current errors, /prompt for your coding standards.

Comprehensive Context (Architecture)

Include multiple files via /file or /tab, load architecture documentation via /fetch, and load your team's conventions via /prompt. Build the context explicitly and review it before asking complex questions.

Advanced Patterns

The Multi-File Context Pattern

For changes that span multiple files:

/file src/models/user.ts
/file src/services/userService.ts  
/file src/routes/users.ts
/file tests/services/userService.test.ts

Add a "preferences" field to the User model and propagate it through the service layer, API routes, and tests.

The Diagnostic-Driven Fix Pattern

Run your build or test suite
Open the assistant panel
/diagnostics to load all current errors
Ask the AI to fix the errors systematically

The Collaborative AI Pattern

Zed's multiplayer features mean multiple developers can collaborate in real time while using AI. One developer can set up the context (load files, configure the prompt) while another reviews the AI's output. This collaborative workflow is unique to Zed and makes it particularly effective for pair programming with AI assistance.

The Speed-Focused Workflow

For developers who prioritize responsiveness:

Use Ollama with a fast local model for inline transformations
Use a cloud model for assistant panel conversations that need more capability
Keep assistant conversations focused and short
Use inline transformations for most edits, reserving the panel for complex tasks

Common Mistakes

Over-including context. Zed gives you explicit control over context. Use it wisely. Including every file in your project via /tab when only 2 files are relevant dilutes the AI's focus.
Not using saved prompts. If you repeat the same instructions across conversations, save them as prompts. One /prompt code-review is better than retyping your review criteria every time.
Ignoring /diagnostics. This command provides structured error context that is faster and more accurate than manually pasting error messages.
Using the assistant panel for simple edits. Inline transformations are faster and require less context setup. Use the panel for complex, multi-file work.
Not exploring provider options. If response quality is not meeting expectations, try a different model. Zed's multi-provider support makes switching easy.
Forgetting /fetch for documentation. External docs can be pulled directly into context without leaving the editor. This is faster and more reliable than manually copying content.

Go Deeper

To learn more about AI-assisted development and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Windsurf: A Complete Guide to the AI Flow IDE

Alex Merced — Sat, 07 Mar 2026 22:00:00 GMT

Windsurf is an AI-powered IDE built on the VS Code foundation that introduces the concept of "Flows," a paradigm where the AI maintains deep awareness of your actions, codebase, and development patterns over time. Its context management differentiates from other editors through Cascade (its agentic coding assistant), persistent Rules files, Memories, and a sophisticated context engine that tracks not just what files you are editing, but how you work.

This guide covers every context management mechanism in Windsurf and explains how to configure them for the most productive development experience.

How Windsurf Manages Context

Windsurf assembles context through multiple layers:

Cascade context engine - tracks your edits, terminal commands, and navigation patterns in real time
Rules files - project and global instructions that shape AI behavior
Memories - persistent facts that carry across sessions
Workspace index - semantic index of your codebase
Conversation context - the current chat session in Cascade
Active editor state - the file you are editing, your cursor position, selected text
MCP server connections - external tools and data sources

The "Flows" concept means Windsurf's AI is not just responding to individual prompts. It maintains a continuous understanding of what you are doing, which enables more relevant suggestions and fewer context-setting instructions from you.

Rules Files: Persistent Project Instructions

Windsurf uses Rules files to define project-level and global instructions for the AI.

Global Rules

Set in Windsurf Settings under AI > Rules, global rules apply across all projects:

# Global Rules

## My Preferences
- Always use TypeScript over JavaScript
- Prefer functional programming patterns
- Use descriptive variable names (no single-letter variables except in loops)
- Add JSDoc comments to all exported functions

## Communication Style
- Be direct and concise
- Show code changes as diffs when possible
- Explain non-obvious design decisions

Project Rules (Workspace)

Create project-level rules in your workspace:

.windsurfrules file in the project root
Or .windsurf/rules/ directory with multiple rule files

# Project: E-Commerce Platform

## Stack
- Next.js 15 with App Router
- TypeScript 5.6
- PostgreSQL with Prisma ORM
- Tailwind CSS 4
- Vitest for testing

## Architecture
- app/ contains page routes and layouts
- lib/ contains shared utilities and API clients
- components/ contains UI components (Atomic Design: atoms, molecules, organisms)
- prisma/ contains schema and migrations

## Conventions
- Server Components by default, Client Components only when necessary
- Use Zod for all input validation
- API routes use the route handler pattern with error boundaries
- All database queries go through Prisma transactions for writes

## Testing
- Every new component needs a unit test
- API routes need integration tests with a test database
- Use MSW for mocking external API calls

Rules Best Practices

Put rules files in version control so the entire team follows the same conventions
Keep rules actionable and specific, not aspirational
Include negative constraints ("Do not use inline styles")
Update rules when you change frameworks, libraries, or conventions
Separate global preferences from project-specific rules

Memories: Persistent Knowledge

Windsurf's Memory system stores facts that persist across conversations and sessions. Memories can be created automatically (when the AI identifies important information during a conversation) or manually.

How Memories Work

When you share something important in a conversation ("We decided to switch from REST to GraphQL for the new API"), Windsurf can save this as a Memory. In future sessions, the AI loads relevant Memories to maintain continuity.

Managing Memories

View all Memories in Windsurf Settings
Delete outdated Memories that no longer apply
Manually add Memories for important decisions the AI should always remember
Review periodically to keep the memory store accurate

Memories vs. Rules

Aspect	Rules	Memories
Creation	You write them explicitly	Created during conversations or manually
Scope	Global or project-level	Cross-project
Purpose	Define conventions and constraints	Store facts and decisions
Update frequency	When conventions change	As new decisions are made

Use Rules for standards and conventions. Use Memories for facts and decisions.

Cascade: The Agentic AI Assistant

Cascade is Windsurf's agentic coding assistant. It operates in two modes with different context management implications:

Chat Mode

Standard conversational interaction where you ask questions and receive answers. Context includes the active file, conversation history, and any files you reference.

Agent Mode

Autonomous mode where Cascade plans and executes multi-step tasks. In Agent Mode, Cascade:

Reads and writes files across your project
Runs terminal commands
Navigates and explores the codebase
Creates and executes multi-file changes

Agent Mode benefits from more comprehensive context (Rules, Memories, workspace index) because it operates autonomously without constant guidance.

MCP Server Support

Windsurf supports MCP for connecting to external tools and data sources.

Configuration

Configure MCP servers through Windsurf Settings or in a configuration file:

{
  "mcpServers": {
    "database": {
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-server-postgres"],
      "env": {
        "DATABASE_URL": "postgresql://dev@localhost:5432/mydb"
      }
    }
  }
}

When to Use MCP

Use MCP in Windsurf for the same scenarios as other IDE-based tools: database queries, GitHub integration, API testing, and browser automation. The integration is seamless because MCP tools become available within Cascade's agent mode.

Model Selection and Context Configuration

Windsurf supports multiple AI providers and models. Your model choice affects context management because different models handle different context window sizes and reasoning capabilities.

Configuring the AI Provider

In Windsurf Settings, you can select from multiple providers:

Windsurf's own models (optimized for the Windsurf context system)
Anthropic (Claude Sonnet, Opus)
OpenAI (GPT-4o, o3)
Custom endpoints (any OpenAI-compatible API)

For complex refactoring that touches many files, choose a model with a larger context window. For quick completions and small edits, a faster model with a smaller window is more responsive.

Tab Completion Context

Windsurf's Tab completion (inline autocomplete) uses a separate context pipeline from Cascade. The completion context includes:

The current file content
Recently edited files
Import statements and type definitions
Patterns from your codebase

Understanding this separation matters because Tab completions are optimized for speed (low latency) while Cascade chat is optimized for depth (comprehensive reasoning). The context for each is assembled differently to match their respective use cases.

How Windsurf Assembles Context

When you interact with Cascade, Windsurf assembles context through this pipeline:

Load Rules: Global rules first, then project rules from .windsurfrules
Load Memories: Retrieve relevant persistent facts
Include active editor state: Current file, cursor position, selection
Process @-commands: Add referenced files, codebase search results, web results
Add flow context: Recent edits, terminal output, navigation patterns
Apply model constraints: Trim to fit within the model's context window

This pipeline runs automatically for every interaction. The more you invest in Rules and Memories, the more relevant the automatically assembled context becomes.

Onboarding a New Project to Windsurf

Here is a step-by-step process for setting up effective context management on a new project:

Day 1: Foundation

Open the project in Windsurf and let the workspace indexing complete
Create a .windsurfrules file with your stack, architecture, and conventions
Make a few small changes to verify Windsurf follows your conventions

Day 2: Refinement

Review what Memories Windsurf created from Day 1
Add any important project facts as manual Memories
Adjust Rules based on how Cascade behaved on Day 1

Week 2: Advanced Setup

Connect relevant MCP servers (database, GitHub)
Index external documentation for @docs references
Start using Agent Mode for multi-file changes
Create directory-specific rules if different modules have different conventions

Thinking About Context Levels

Quick Edits (Minimal Context)

Use inline editing (Cmd+K / Ctrl+K) for small changes. Windsurf uses the current file and selection, plus applicable Rules, to generate edits. No additional context needed.

Feature Development (Moderate Context)

Use Cascade chat with explicit file references. The workspace index, Rules, and Memories combine to give Cascade project-aware responses.

Complex Architecture Work (Comprehensive Context)

Use Agent Mode with well-configured Rules, active Memories, and MCP connections. Let Cascade explore the codebase, run commands, and make changes across multiple files.

@ Commands for Context Injection

Windsurf supports @-commands similar to Cursor for injecting specific context:

@file - Reference a specific file
@codebase - Search the indexed codebase
@web - Search the web for current information
@docs - Reference indexed documentation
@terminal - Include terminal output context

These commands give you fine-grained control over what context Cascade receives for each prompt.

External Documents: PDFs vs. Markdown

Markdown for Rules

All Rules files and project documentation should be Markdown. It is the native format for Windsurf's context system.

For Reference Material

For external specifications in PDF form, convert key sections to Markdown and include them in your project as reference documents. This makes them discoverable through @codebase searches.

Documentation Indexing

Like Cursor, Windsurf can index external documentation. Add framework and library docs to the indexed sources so @docs references return relevant, up-to-date information.

Advanced Patterns

The Flow-Aware Development Pattern

Leverage Windsurf's flow tracking by working naturally:

Make edits in the editor (Windsurf tracks your changes)
Run tests in the terminal (Windsurf observes the results)
Ask Cascade a question (it already knows what you changed and what failed)

This removes the need to manually explain what you just did. Windsurf already knows.

The Rules-Layered Workflow

Combine global and project rules for comprehensive coverage:

Global rules: Your personal coding style and preferences
Project rules: Team conventions and architecture decisions
Directory-specific rules: Module-specific patterns

The Agent-Then-Review Pattern

Describe the feature to Cascade in Agent Mode
Let it plan and implement the changes
Review each file change in the diff view
Accept, reject, or modify individual changes
Ask Cascade to adjust based on your feedback

This uses Agent Mode for speed while maintaining human oversight through the review step.

The Memory-Driven Continuity Pattern

At the end of each working session, review what Windsurf has stored as Memories. Add any important decisions or discoveries that were not automatically captured. At the start of the next session, Cascade starts with a richer understanding of your project.

Common Mistakes

Not setting up Rules files. Without them, Cascade applies generic conventions. Project-specific Rules are the highest-impact configuration.
Ignoring Memories. Stale Memories mislead the AI. Review and clean them periodically.
Underusing Agent Mode. For multi-file changes, Agent Mode is dramatically faster than chat-based interactions. Trust it for structural changes and review the results.
Over-specifying context in prompts. If your Rules and Memories are well-configured, you do not need to re-explain your conventions in every prompt.
Not leveraging flow awareness. Windsurf tracks your actions. Instead of explaining what you just did, ask questions that build on your recent work.
Skipping @codebase for exploration. When you are unsure which files are relevant, @codebase search is more efficient than manually navigating the project tree.

Go Deeper

To learn more about AI-assisted development and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Perplexity AI: A Complete Guide to Research-First AI Conversations

Alex Merced — Sat, 07 Mar 2026 21:00:00 GMT

Perplexity AI occupies a unique position in the AI landscape: it is a research-first tool that combines conversational AI with real-time web search to produce answers grounded in current sources. Unlike coding-focused tools or general chatbots, Perplexity is built for information retrieval, analysis, and synthesis. Its context management is designed around Spaces (persistent research workspaces), Focus Modes (search scope control), and an elastic context window that adapts to the complexity of your query.

This guide covers how to manage context effectively in Perplexity for everything from quick fact-checking to sustained research projects.

How Perplexity Manages Context

Perplexity builds context from several sources:

Web search results - real-time retrieval of current information
Spaces - persistent workspaces with uploaded files and custom instructions
Focus Modes - filters that control which sources are searched
Conversation history - the thread of questions and answers in the current session
Uploaded files - documents you provide for analysis
Memory - persistent facts the system remembers about you (enterprise plans)

The key difference from other AI tools is that Perplexity actively searches the web for every query by default. This means its context combines your instructions and uploaded files with fresh, real-time information from the internet, producing answers with citations that you can verify.

Spaces: Persistent Research Workspaces

Spaces are Perplexity's equivalent of Projects in other tools. A Space groups related conversations, files, and instructions into a persistent workspace.

Creating a Space

Navigate to Spaces in the sidebar
Create a new Space with a descriptive name
Add Custom Instructions: Guidelines that shape every response in this Space
Upload files: PDFs, documents, spreadsheets, and other reference material
Set the default Focus Mode for the Space

Space Instructions

Instructions in a Space function like a system prompt for every conversation within it:

# Research Space: Renewable Energy Markets

## Role
You are a market research assistant focused on renewable energy.

## Requirements
- Cite all claims with sources less than 6 months old
- Include market size and growth rate data when available
- Compare data across geographic regions when relevant
- Flag any statistics from sources over 1 year old

## Format
- Use structured sections with clear headers
- Include a "Sources" section at the end of every response
- Present data in tables when comparing multiple items

File Uploads in Spaces

Spaces support various file types for persistent reference:

File Type	Use Case
PDF	Research papers, reports, whitepapers
Documents	Analysis templates, style guides
Spreadsheets	Data for analysis and comparison
Text/Markdown	Notes, outlines, custom context documents

Files in a Space are available across all conversations in that Space. This means you upload a report once and can reference it in every subsequent conversation.

When to Create a Space

You are researching a topic across multiple sessions
You have reference documents you want the AI to consult alongside web results
You need consistent response formatting and focus
You are working on a project that requires accumulating research over time

Focus Modes: Controlling Search Scope

Focus Modes let you control where Perplexity searches for information:

Focus Mode	Sources	Best For
All	Entire web	General research, broad questions
Academic	Google Scholar, research databases	Scientific research, literature reviews
Writing	No web search (uses training data)	Content creation, drafting, brainstorming
Math	Computation-focused, mathematical sources	Calculations, proofs, statistical analysis
Video	YouTube and video platforms	Tutorial discovery, visual explanations
Social	Reddit, forums, social platforms	Community opinions, user experiences, discussions

Using Focus Modes as Context Filters

Focus Modes are a form of context management because they determine what kind of information reaches the model. Choosing the right Focus Mode prevents irrelevant results from diluting the response:

Researching a technical specification? Use "All" for comprehensive coverage
Writing a literature review? Use "Academic" to prioritize peer-reviewed sources
Looking for real-world experiences? Use "Social" to surface personal accounts and community discussions
Drafting text without needing web data? Use "Writing" to focus on generation rather than retrieval

Switching Focus Modes Mid-Research

You can switch Focus Modes within a Space. A common pattern:

Start with "Academic" to find foundational research
Switch to "All" for industry reports and market data
Use "Social" to gauge public perception and user experiences
Switch to "Writing" to draft your synthesis

Each mode shapes the context differently, giving you control over the type of information the model works with.

Thinking About Context Levels

Quick Questions (Minimal Context)

For factual questions with clear answers, just ask. Perplexity will search the web and return a sourced response:

"What is the current market size of the global data analytics industry?"

No Space, no file uploads, no special Focus Mode needed. Perplexity's default behavior handles this well.

Focused Research (Moderate Context)

For deeper exploration, create a Space with instructions and upload relevant reference material:

"Based on the market report I uploaded and current web data, compare the growth trajectories of the three largest cloud providers in the data analytics space."

The combination of uploaded files (for baseline data) and web search (for current information) produces comprehensive analysis.

Extended Research Projects (Comprehensive Context)

For multi-week research projects, use a fully configured Space with detailed instructions, multiple uploaded documents, and strategic Focus Mode switching. Build on previous conversations by referencing insights from earlier threads.

Deep Research 2.0

Perplexity's Deep Research feature performs multi-step research autonomously. When you invoke Deep Research, the system:

Analyzes your question and creates a research plan
Executes multiple web searches across diverse sources
Reads and analyzes full articles (not just snippets)
Synthesizes findings into a comprehensive report
Provides structured output with citations for every claim

Deep Research is available on Pro plans and uses significantly more compute than standard queries. The tradeoff is worth it for complex questions that require multi-source synthesis.

Context Management for Deep Research

Deep Research benefits from clear, specific prompts. Because the system executes autonomously, your initial prompt is the primary context it works from:

Less effective: "Tell me about AI in healthcare"

More effective: "Research the current state of AI-powered diagnostic tools in radiology. Focus on: (1) FDA-approved systems as of 2026, (2) clinical accuracy compared to human radiologists, (3) adoption rates across US hospitals, and (4) barriers to wider adoption. Prioritize peer-reviewed sources and official regulatory data."

The specific prompt gives Deep Research a structured plan to follow, producing a more focused and useful report.

Structuring Prompts for Effective Context

The Research Question Framework

Structure your prompts using this framework for best results:

Topic: What are you researching?
Scope: What specific aspects matter?
Sources: What type of sources do you want?
Recency: How current must the information be?
Format: How should the response be structured?

Example: "Research [topic]. Focus on [scope]. Prioritize [source type] from [time period]. Present findings as [format]."

Follow-Up Strategies

Perplexity maintains conversation context within a thread. Use follow-ups strategically:

Drilling down: "Tell me more about point 3 from your previous response"
Pivoting: "How does this compare to the European market?"
Validating: "Find additional sources that support or contradict the statistics you cited"
Updating: "What has changed on this topic in the last 3 months?"

Each follow-up builds on the accumulated context of the conversation, producing progressively deeper analysis.

MCP Server Support

Perplexity supports local MCP (Model Context Protocol) servers on its macOS desktop application. This allows the AI to connect to external tools and data sources running on your local machine, extending its capabilities beyond web search.

How MCP Works in Perplexity

On the macOS app, you can configure local MCP servers that provide Perplexity with access to your file system, local databases, applications, and other services. This is configured through the app's settings. Remote MCP servers (cloud-based services) are planned for paid subscribers.

When MCP Adds Value

For most Perplexity use cases, web search is the primary context extension mechanism. MCP adds value when you need Perplexity to combine its web research capabilities with local data:

Researching a topic while cross-referencing your local documents
Analyzing data from a local database alongside web-sourced information
Integrating with local development tools or APIs

Web Search vs. MCP

Where other tools use MCP to reach databases or APIs, Perplexity's distinguishing feature is its web search capability. MCP complements this by adding local data access, but for most research workflows, Perplexity's web search provides the primary context extension. If you need extensive MCP functionality (writing code, managing databases, interacting with multiple external services), pair Perplexity with a coding-focused tool like Claude Desktop or Cursor.

External Documents: PDFs vs. Markdown

PDFs in Perplexity

Perplexity handles PDFs well, especially for research papers and reports. Upload them to a Space for persistent reference. Perplexity can:

Extract text and answer questions about the content
Compare information across multiple uploaded PDFs
Combine uploaded PDF data with web search results

Markdown

For context documents you author (instructions, outlines, research frameworks), Markdown is cleaner and more precisely parsed. Use Markdown for structure-dependent content where formatting matters.

The Hybrid Approach

Use PDFs for received documents (research papers, reports, specifications). Use Markdown for documents you create (Space instructions, research frameworks, output templates).

Memory (Enterprise)

On enterprise plans, Perplexity supports persistent Memory that remembers facts about you across conversations. This is similar to ChatGPT's Memory feature and stores preferences, role information, and recurring context that you should not have to re-state every time.

For individual users, Spaces serve a similar purpose by maintaining per-workspace instructions and files, even though the memory mechanism is different.

Advanced Patterns

The Research Pipeline Pattern

Use Perplexity as the front end of a research pipeline:

Discovery: Use Deep Research to survey a topic comprehensively
Validation: Switch to Academic Focus to verify key claims with peer-reviewed sources
Community insight: Switch to Social Focus to understand real-world adoption and reception
Synthesis: Switch to Writing Focus to draft your analysis based on the accumulated context
Export: Copy the synthesized research into your writing tool of choice

The Comparative Analysis Pattern

Use Spaces to compare multiple topics or options:

Upload comparison criteria as a Markdown file
Research each option in a separate conversation within the Space
Use a final conversation to synthesize the findings into a comparison table

The Space maintains the criteria and accumulated research across all conversations.

The Source Quality Verification Pattern

Use Focus Mode switching to verify claims across different source types:

Find a claim in "All" mode
Verify it in "Academic" mode (peer-reviewed backing)
Check reception in "Social" mode (how practitioners view the claim)
Check for retractions or updates in "All" mode with a date filter

This multi-angle verification produces higher-confidence research than relying on a single source type.

Common Mistakes

Not using Spaces for project research. Individual conversations lose context when you close them. Spaces maintain your instructions, files, and conversation history persistently.
Ignoring Focus Modes. Using "All" for everything misses the specialized results that Academic, Social, and other modes provide. Match the mode to the question.
Vague Deep Research prompts. Deep Research executes autonomously, so a vague prompt produces a vague report. Be specific about what you want investigated and how you want it structured.
Uploading too many unrelated files to one Space. Keep Spaces focused on specific topics. A Space with 30 unrelated documents dilutes the context for any specific query.
Not verifying citations. Perplexity provides source citations for a reason. Click through and verify key claims, especially for high-stakes research.
Using Perplexity for tasks that need code execution or local tool access. Perplexity is a research tool, not a coding agent. For tasks requiring code execution, terminal access, or database interaction, use a coding-focused tool instead.

Go Deeper

To learn more about AI-assisted research, context management, and agentic workflows, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Cursor: A Complete Guide to the AI-Native Code Editor

Alex Merced — Sat, 07 Mar 2026 20:00:00 GMT

Cursor is an AI-native code editor built on the VS Code foundation that integrates AI deeply into every aspect of the development workflow. Its context management system is one of the most sophisticated among coding tools, combining workspace-level indexing, granular rules files, documentation integration, MCP server support, and intelligent context assembly that automatically determines which files and symbols are relevant to your current task.

This guide covers every context management mechanism Cursor provides and explains how to configure them for productive, reliable AI-assisted development.

How Cursor Manages Context

Cursor assembles context from multiple sources, with intelligent prioritization:

Workspace index - a semantic index of your entire codebase built on first open
.cursor/rules/ files - project-specific instructions in MDC format
@-mentions - explicit context you inject into prompts (@file, @codebase, @Docs)
MCP server connections - external tools and data
Active file and selection - the code you are currently looking at
Conversation history - recent messages in the current chat session
Debug context - error messages, stack traces, and terminal output

The workspace index is what makes Cursor's context management stand out. Instead of relying on you to specify which files are relevant, Cursor semantically indexes your entire project and retrieves the most relevant code based on your query.

.cursor/rules/: Project-Level Instructions

Cursor uses .cursor/rules/ files in MDC (Markdown Configuration) format to provide project-level instructions. These files tell Cursor how to behave within your project.

Rule Types

Type	Behavior	Best For
Always	Loaded for every interaction	Core conventions, style preferences
Auto	Loaded when matched files are active	File-type specific rules (e.g., Python vs. TypeScript)
Agent	Available to the agent for self-selection	Specialized knowledge the agent invokes when needed
Manual	Only loaded when explicitly referenced	Rarely used instructions you invoke for specific tasks

Creating Rules

Create .mdc files in .cursor/rules/:

---
description: Python coding standards for this project
globs: ["**/*.py"]
alwaysApply: false
---

# Python Rules

## Style
- Use type hints for all function parameters and return values
- Use dataclasses or Pydantic models instead of plain dicts
- Prefer f-strings over .format() or %-formatting
- Maximum line length is 88 characters (Black default)

## Testing
- Use pytest, not unittest
- Test files mirror the source tree: src/services/auth.py -> tests/services/test_auth.py
- Use factories for test data, not fixtures
- Mock external services at the client boundary

## Architecture
- Business logic lives in src/services/
- Database access goes through src/repositories/
- API routes are thin: validate input, call service, return response
- Never import from internal modules; use the package's public API

Rules Best Practices

Use globs to target rules. Auto rules with specific glob patterns (like **/*.py) keep Python conventions separate from JavaScript conventions.
Keep rules actionable. Every rule should describe a specific behavior the agent should follow. Vague guidance like "write clean code" wastes tokens.
Document your architecture. Tell Cursor where things live. Understanding your project structure prevents the agent from putting code in the wrong place.
Include negative constraints. "Do NOT use class-based views" is often more effective than a long description of what to use instead.

@-Mentions: Explicit Context Injection

Cursor's @-mention system lets you add specific context to any prompt.

Available @-Mentions

Mention	Purpose
`@file`	Reference a specific file by name
`@codebase`	Search the entire indexed codebase for relevant context
`@Docs`	Search indexed documentation
`@web`	Search the web for current information
`@git`	Reference Git history (diffs, commits, branches)
`@definitions`	Include symbol definitions referenced in your selection
`@folders`	Include directory structure context

Using @codebase Effectively

@codebase is the most powerful @-mention because it triggers semantic search across your entire project. When you type:

"@codebase How is authentication implemented in this project?"

Cursor searches its semantic index, retrieves the most relevant files and symbols, and includes them in the context. This is far more efficient than manually specifying each file.

@Docs: Documentation-Aware Context

You can index external documentation sources so Cursor can reference them:

Go to Cursor Settings > Features > Docs
Add documentation URLs (framework docs, API references, internal wikis)
Cursor crawls and indexes the documentation
Use @Docs in prompts to reference the indexed content

Example: "Using @Docs for React 19, refactor this component to use the new use() hook."

This is particularly valuable for newer libraries where the AI's training data may be outdated.

MCP Server Support

Cursor supports MCP for connecting to external tools and services.

Configuration

MCP servers are configured in Cursor's settings:

{
  "mcpServers": {
    "postgres": {
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-server-postgres"],
      "env": {
        "DATABASE_URL": "postgresql://dev@localhost:5432/mydb"
      }
    },
    "github": {
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-server-github"]
    }
  }
}

When to Use MCP in Cursor

MCP is most valuable when the task requires live data from outside the codebase:

Querying a development database to understand schema or verify data
Interacting with GitHub for PR reviews or CI status
Accessing internal APIs to verify integration behavior
Running browser automation to test frontend changes

For code-only tasks (refactoring, writing tests, fixing bugs), Cursor's built-in codebase index is sufficient.

Debug Mode and Error Context

Cursor offers a Debug Mode that automatically provides error context to the AI:

When you encounter an error in the terminal or running application
Cursor captures the error message, stack trace, and relevant file context
You can ask the AI to diagnose and fix the issue with full context

This automatic error context gathering is a significant context management feature because it eliminates the manual process of copying error messages and stack traces into prompts.

Thinking About Context Levels

Minimal Context (Quick Fixes)

For small edits, select code in the editor and use inline editing (Cmd+K / Ctrl+K). Cursor uses the current file and selection as context. No additional setup needed.

Moderate Context (Feature Development)

Use the chat panel with @-mentions. Reference the relevant files with @file, use @codebase for broader understanding, and include @Docs for framework-specific guidance.

Comprehensive Context (Architecture Work)

Combine .cursor/rules/ with @codebase and MCP servers. The rules provide your conventions, @codebase provides structural understanding, and MCP provides live system context.

External Documents: PDFs vs Markdown

Markdown for Rules

All .cursor/rules/ files use MDC (Markdown-based) format. Your coding standards, style guides, and architectural documentation should be in this format.

Documentation Indexing

For external documentation, use the @Docs system to index web-based docs directly. This is more effective than converting PDFs to Markdown because Cursor handles the indexing and retrieval automatically.

For Reference Material

If you have specifications or design documents in PDF form, the most practical approach is to extract key sections into .mdc rule files or Markdown documents in your repository. This makes them searchable through @codebase.

Model Selection and Context Windows

Cursor supports multiple AI providers and models. Your model choice affects context management because different models have different context window sizes and capabilities.

Context Window Considerations

Model	Context Window	Best For
Claude Sonnet	200K tokens	Large codebase analysis, complex refactoring
GPT-4o	128K tokens	Feature development, code generation
Cursor Small	Varies	Quick edits, inline completions

For large projects, choose a model with a bigger context window so Cursor can include more codebase context without hitting limits. For simple edits, a smaller, faster model is more responsive.

How Cursor Assembles Context

When you send a message in Cursor's chat, the editor automatically assembles context by:

Including the active file and your cursor position
Including any @-mentioned files or resources
Searching the workspace index if @codebase is used
Loading applicable rules from .cursor/rules/ based on the active file type
Including recent conversation history for continuity
Adding any MCP server tool descriptions for agent mode

This automatic assembly is why Cursor often produces better results than manually pasting code into a generic chatbot. The context is structured and relevant, not random.

Context Budget Management

Each prompt has a context budget limited by the model's context window. When the budget is tight:

Be selective with @file mentions (reference only files directly relevant to the task)
Use @codebase instead of @file for exploratory questions (it retrieves only relevant snippets)
Keep rules files concise and targeted
Start new chat sessions when switching topics

Workspace Indexing Deep Dive

The workspace index is Cursor's most powerful context feature. It creates a semantic understanding of your entire codebase that powers @codebase searches and the agent's ability to navigate your project.

How Indexing Works

When you open a project in Cursor:

Cursor scans all files (respecting .gitignore)
It creates embeddings (semantic representations) of code symbols, functions, and classes
These embeddings are stored in a local index
When you ask questions, Cursor searches this index for the most relevant code

Indexing Best Practices

Let the index complete before starting work. Look for the indexing indicator in the status bar.
Re-index after major changes. If you merge a large branch or restructure directories, trigger a re-index.
Trust the index. @codebase search often finds more relevant code than you would think to include manually.

Practical Workflow Recommendations

For New Projects

Open the project in Cursor and let it index
Create .cursor/rules/ with your core coding standards
Add @Docs entries for the frameworks you are using
Start with small tasks to verify Cursor understands your conventions

For Team Adoption

Check .cursor/rules/ into version control
Agree on shared rule categories: Always rules for team-wide standards, Auto rules for language-specific patterns
Add team documentation to @Docs
Create Agent rules for specialized knowledge (deployment, database conventions)

For Complex Features

Start with @codebase to understand the existing implementation
Use Composer for multi-file changes
Reference @Docs for framework-specific guidance
Use Debug Mode to quickly resolve implementation issues

Advanced Patterns

The Notepads Pattern

Cursor's Notepads feature lets you create persistent context documents within the editor. Unlike .cursor/rules/ (which are loaded automatically), Notepads are reference documents you can @-mention when needed:

Architecture decision records
API specifications
Design system documentation
Onboarding guides for new team members

The Composer Pattern

Use Cursor's Composer (multi-file agent mode) for changes that span multiple files:

Describe the feature or change you want
Composer plans modifications across relevant files
Review the proposed changes
Apply or reject each file modification individually

Composer automatically assembles context from the workspace index, making it effective for cross-cutting changes.

The Rules Layering Strategy

Combine different rule types for comprehensive coverage:

Always rules: Universal team conventions (style, testing, documentation)
Auto rules: Language-specific standards (Python patterns, TypeScript patterns)
Agent rules: Specialized knowledge (deployment procedures, database conventions)

This layering ensures the right context is active for the right task without overloading every interaction.

Common Mistakes

Not creating .cursor/rules/. Without rules, Cursor applies generic conventions that may not match your project. The rules are the single highest-impact configuration you can make.
Ignoring @codebase. Many users manually specify files when @codebase would find the relevant code automatically. Trust the semantic search.
Not indexing documentation. If you are using a newer framework, @Docs with indexed documentation prevents the AI from relying on outdated training data.
Over-specifying context. If you include 20 files via @file when only 3 are relevant, you dilute the AI's attention. Use @codebase to let Cursor find the right files, or be selective with @file mentions.
Skipping the workspace indexing. Let Cursor finish indexing your workspace on first open. The index powers @codebase and context assembly. Without it, context quality degrades significantly.
Not using Debug Mode. When errors occur, Debug Mode provides structured error context that significantly improves the AI's diagnostic accuracy compared to manually pasting error messages.

Go Deeper

To learn more about AI-assisted development and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for OpenWork: A Complete Guide to the Desktop AI Agent Framework

Alex Merced — Sat, 07 Mar 2026 19:00:00 GMT

OpenWork is a desktop-native AI agent framework designed for local, multi-step task execution on your computer. Unlike browser-based AI tools or terminal agents, OpenWork operates as a desktop application that can interact with your file system, manage long-running sessions, and execute complex workflows autonomously. Its context management centers on Skills, session persistence, direct file system access, and a plugin architecture that extends its capabilities.

This guide explains how to manage context effectively in OpenWork to delegate complex tasks, maintain continuity across sessions, and build reusable automation workflows.

How OpenWork Manages Context

OpenWork builds its context from several layers:

Skills - predefined capability packages that define what the agent can do
Session state - persistent history and progress tracking across interactions
File system access - direct read/write access to your local files
Plugin extensions - additional capabilities including MCP server connections
Task definitions - structured descriptions of multi-step workflows
Your instructions - natural language guidance provided at task creation

The key difference between OpenWork and other tools is its desktop-native design. It is built to interact with your operating system, not just with text in a terminal or browser. This means it can manage files, organize folders, process documents, and perform tasks that span multiple applications.

Skills: The Foundation of OpenWork's Capabilities

Skills in OpenWork define focused areas of expertise. Each Skill packages instructions, tools, and workflows into a reusable unit.

Built-In Skills

OpenWork ships with core Skills for common tasks:

File Management: Organizing, renaming, moving, and transforming files
Document Processing: Reading, summarizing, and creating documents
Data Analysis: Processing spreadsheets, CSVs, and structured data
Web Research: Gathering information from web sources
Code Assistance: Writing, reviewing, and refactoring code

Creating Custom Skills

Define custom Skills that match your specific workflows:

# Skill: Monthly Report Generator

## Purpose
Generate monthly departmental reports by combining data from multiple sources.

## Inputs Required
- Sales data CSV from /data/sales/
- Customer feedback file from /data/feedback/
- Team metrics from /data/team/

## Process
1. Read and validate all input files
2. Calculate key metrics (revenue, growth, satisfaction scores)
3. Generate narrative summary for each section
4. Format the report using the template in /templates/monthly-report.md
5. Save to /reports/YYYY-MM-monthly-report.md

## Quality Checks
- All numerical values must be sourced from the input data
- The report must include year-over-year comparisons
- Format all currency values with two decimal places

Skill Selection and Context

When you assign a task, OpenWork selects the relevant Skills based on the task description. The selected Skills become part of the active context, giving the agent the specific instructions it needs for that type of work. This means well-defined Skills reduce the amount of context you need to provide in each task description.

Session Management and Persistence

OpenWork maintains persistent sessions that carry context across interactions. This is critical for multi-step tasks that span hours or days. Unlike web-based AI tools where closing the browser tab loses your conversation state, OpenWork sessions are durably stored on your local machine and survive application restarts.

Session State

Each session tracks:

Conversation history: Every instruction and response
File operations: What files were read, created, or modified
Task progress: Current step in multi-step workflows
Agent decisions: Why specific actions were taken (for auditability)

Resuming Sessions

When you return to OpenWork after closing it, your sessions are preserved. You can:

Continue where you left off on an interrupted task
Review what the agent did while you were away (for scheduled tasks)
Provide additional instructions based on completed work

Starting Fresh

For unrelated work, start a new session. Carrying over context from a previous project creates noise that degrades the agent's focus.

File System Access: Direct Local Interaction

OpenWork's direct file system access is one of its primary context advantages. The agent reads files in real time (not from uploaded snapshots) and writes output directly to your file system.

Context from Your File System

Project structures: The agent can browse directories to understand organization
Document contents: Read any text-based file without manual copying
Data files: Process CSVs, JSON files, and other structured data in place
Configuration: Read settings files to understand tool configurations

Best Practices

Organize files before delegating. A well-structured file system gives OpenWork better context than a messy one.
Use descriptive file names. q3-revenue-analysis.csv gives the agent more context than data2.csv.
Create a dedicated working directory for each project or task category.
Store templates in a consistent location so Skills can reference them.

MCP Support Through Plugins

OpenWork supports MCP servers through its plugin architecture, enabling connections to external data sources and tools.

When MCP Adds Value

Database integration: Let OpenWork query databases for report generation
Cloud storage: Access files in Google Drive, OneDrive, or S3
API integration: Connect to internal services for data retrieval
Communication tools: Draft messages or pull context from Slack, email, or other platforms

Configuration

MCP servers are configured through OpenWork's settings panel. Each server connection becomes available as a tool that Skills can utilize.

Thinking About Context Levels

Simple Tasks (Minimal Context)

For straightforward file operations ("Rename all files in /downloads/ to include today's date"), the task description and file system access provide sufficient context.

Moderate Tasks

For tasks requiring judgment ("Review the documents in /contracts/ and flag any that expire within 30 days"), provide the criteria and desired output format. OpenWork will use its Skills and file access to execute.

Complex Tasks

For multi-step workflows ("Create a quarterly business review presentation from data in three different folders, following the template in /templates/"), invest in a detailed task definition and ensure the relevant Skills are configured.

Structuring Context for Effective Delegation

The Briefing Document Approach

For complex tasks, create a briefing document that OpenWork reads before starting:

# Task Briefing: Q3 Performance Analysis

## Objective
Create a comprehensive performance analysis comparing Q3 results 
against Q2 and the same quarter last year.

## Data Sources
- /data/revenue/q3-2026.csv (primary revenue data)
- /data/revenue/q2-2026.csv (previous quarter)
- /data/revenue/q3-2025.csv (year-over-year comparison)
- /data/kpis/team-metrics.json (operational metrics)

## Required Sections
1. Executive Summary (250 words max)
2. Revenue Analysis with charts
3. Year-over-Year Comparison
4. Team Performance Metrics
5. Recommendations

## Formatting
- Use the template at /templates/quarterly-analysis.md
- All percentages to one decimal place
- Currency in USD with comma separators
- Charts as ASCII/text-based tables

## Quality Standards
- Every claim must reference a specific data point
- Include both absolute and percentage change figures
- Flag any anomalies or data gaps

This structured briefing gives OpenWork comprehensive context without relying on interactive conversation.

The Progressive Detail Pattern

Provide context in layers, starting broad and getting specific:

High-level goal: "Create a monthly financial report"
Specific requirements: "Include revenue, costs, and margin analysis"
Data locations: "Source data is in /finance/monthly/"
Quality criteria: "All numbers must reconcile with the source data"
Output format: "Follow the template in /templates/"

Each layer adds specificity without contradicting previous layers.

Multi-Agent Coordination

OpenWork can coordinate multiple agents working on related but independent tasks:

Parallel Execution

Agent 1: Processes financial data and creates charts
Agent 2: Summarizes customer feedback from text files
Agent 3: Compiles operational metrics from log files

Each agent works with its own focused context, and the results are combined into a final deliverable.

Sequential Handoffs

For workflows where each step depends on the previous one:

Agent A produces raw analysis
Agent B reviews and refines the analysis
Agent C formats the final output

The context from each step flows to the next, creating a pipeline of increasingly refined output.

External Documents: PDFs vs. Markdown

PDFs

OpenWork can read PDFs directly from your file system. Use PDFs for:

Existing reports and documents that are already in PDF format
External specifications or contracts received from others
Formatted documents where layout matters

Markdown

For documents you create specifically for OpenWork (templates, instructions, style guides), use Markdown. It parses more reliably and is easier for the agent to reference precisely.

The File-Based Advantage

Because OpenWork accesses files directly (not through uploads), the format matters less than it does for web-based tools. Both PDFs and Markdown are readable from the file system. Choose based on the source: use the original format for received documents, and Markdown for documents you author.

Advanced Patterns

The Scheduled Workflow Pattern

Set up recurring tasks that OpenWork executes on a schedule:

Define the task with clear inputs, processes, and outputs
Schedule it to run at a specific time (daily, weekly, monthly)
OpenWork executes the task autonomously and saves the results
Review the output when convenient

This is ideal for report generation, data processing, file organization, and routine maintenance tasks.

The Multi-Step Pipeline Pattern

Chain multiple Skills into a pipeline:

Step 1 (Data Collection): Gather data from multiple sources
Step 2 (Processing): Clean, transform, and analyze the data
Step 3 (Generation): Create the output document or presentation
Step 4 (Verification): Check the output against quality criteria

Each step builds on the context from previous steps, creating a coherent end-to-end workflow.

The Delegation Escalation Pattern

Start with simple delegations and gradually increase complexity:

Week 1: File organization and simple document creation
Week 2: Data processing and report generation
Week 3: Multi-source research and synthesis
Week 4: Fully automated recurring workflows

This builds your confidence in OpenWork's handling of context while gradually training the agent (through Skills and session history) on your specific needs.

When to Use OpenWork vs. Other Tools

Use OpenWork when:

Your tasks involve desktop-level file management
You need multi-step autonomous execution
You want scheduled, recurring task automation
Your work is document-centric (reports, presentations, data processing)

Use a terminal agent (Claude Code, Gemini CLI, OpenCode) when:

Your work is code-centric
You need direct terminal command execution
You want inline access to compilers, test runners, and build tools

Use a web-based tool (ChatGPT, Claude Web) when:

You need interactive conversation and brainstorming
The task is primarily knowledge-based
You do not need local file system access

Common Mistakes

Vague task descriptions. "Work on my files" gives OpenWork nothing to execute. Specify what files, what action, and what output you expect.
Skipping Skills for repeatable work. If you delegate the same type of task more than twice, create a Skill for it.
Not reviewing autonomous output. Scheduled tasks run without supervision. Always review the results, especially during the first few runs.
Disorganized file systems. OpenWork's effectiveness depends on finding and understanding your files. Messy directories produce messy results.
Over-scoping single tasks. Break large projects into multiple tasks with clear handoff points. OpenWork handles focused, well-defined tasks better than vague, sweeping ones.
Not leveraging session persistence. If a task is partially complete, resume the session rather than starting over. The carried context improves continuity.

Go Deeper

To learn more about working effectively with AI agents and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for OpenCode: A Complete Guide to the Open-Source Terminal AI Agent

Alex Merced — Sat, 07 Mar 2026 18:00:00 GMT

OpenCode is an open-source terminal-based AI coding agent that prioritizes privacy, local-first operation, and broad model provider support. Built as a TUI (terminal user interface) application, it runs entirely in your terminal and supports dozens of LLM providers from OpenAI and Anthropic to local models through Ollama. Its context management system is built around configuration files, session persistence, MCP integration, and a dual-agent architecture that separates planning from code generation.

This guide covers every context management mechanism OpenCode offers and explains how to configure them for effective development workflows, regardless of which model provider you choose.

The TUI Advantage for Context Management

OpenCode's TUI (Terminal User Interface) provides a structured visual interface within your terminal. Unlike bare CLI tools where you interact through plain text, the TUI offers:

A conversation panel showing the full history with syntax-highlighted code blocks
A file browser for navigating your project structure
A status bar showing the active model, session state, and token usage
Visual indicators for agent mode (Plan vs. Build)

The TUI makes context management more tangible because you can see what the agent is working with. Token usage indicators help you understand when you are approaching context limits, and the session panel lets you manage conversation history visually.

How OpenCode Manages Context

OpenCode assembles its context from several sources:

opencode.json - project-level configuration and instructions
Session history - SQLite-backed persistent sessions
MCP server connections - external tools and data
LSP (Language Server Protocol) integration - real-time code intelligence
The codebase - files, directories, and project structure
Custom commands - user-defined reusable operations

What distinguishes OpenCode from other terminal agents is its architectural separation between "Build" and "Plan" agents. The Build agent writes code and makes changes. The Plan agent reasons about architecture and strategy without modifying files. This separation affects how you structure context: planning tasks need architectural context, while building tasks need implementation detail.

opencode.json: Project Configuration

The opencode.json file in your project root is the primary configuration mechanism. It defines provider settings, model selection, and project-specific context.

Basic Configuration

{
  "$schema": "https://opencode.ai/config.schema.json",
  "provider": {
    "name": "anthropic",
    "model": "claude-sonnet-4.5"
  },
  "context": {
    "instructions": "This is a Python FastAPI application with PostgreSQL. Use Ruff for linting and pytest for testing. Follow PEP 8 strictly.",
    "include": ["src/", "tests/", "docs/"],
    "exclude": ["*.pyc", "__pycache__/", ".venv/"]
  }
}

Context Instructions

The context.instructions field functions like CLAUDE.md or GEMINI.md for other tools. Include:

Your technology stack and versions
Coding conventions and style preferences
Testing strategy and framework
Architecture decisions and patterns
Build and deployment commands

Include and Exclude Patterns

Control what OpenCode sees by specifying include and exclude patterns. This focuses the agent's attention on relevant code and prevents it from wasting context on generated files, dependencies, or build artifacts.

Provider Flexibility

OpenCode supports a wide range of providers:

Provider	Models	Notes
OpenAI	GPT-4o, o3, etc.	Cloud-hosted
Anthropic	Claude Sonnet, Opus	Cloud-hosted
Google	Gemini Pro, Flash	Cloud-hosted
Ollama	Llama, Mistral, etc.	Local, private
OpenRouter	Many models	Multi-provider routing
Custom endpoints	Any OpenAI-compatible API	Self-hosted

This flexibility means you can choose the right model for your context needs. Local models through Ollama keep all context on your machine. Cloud models provide more capability but send your context to external servers.

The Dual-Agent Architecture: Build vs. Plan

OpenCode's most distinctive context management feature is its separation of planning and execution into two independent agents.

The Plan Agent

The Plan agent reasons about architecture, strategy, and design without making any file changes. Use it for:

Analyzing a codebase before making changes
Designing an implementation approach
Evaluating tradeoffs between different solutions
Understanding unfamiliar code

The Plan agent receives the same project context (opencode.json, codebase, MCP) but operates in a read-only mode. This is valuable because it means you can explore and discuss ideas without risk of unintended changes.

The Build Agent

The Build agent writes code, creates files, runs commands, and makes changes to your project. It uses the planning context plus implementation-specific details:

The specific files that need modification
Test commands to verify changes
Style and formatting requirements

Switching Between Agents

Switch between Plan and Build during a session to match the current need:

Start with Plan: "Analyze the authentication module and suggest how to add OAuth support"
Review the plan: Evaluate the agent's architectural proposal
Switch to Build: "Implement the OAuth integration following the approach you described"

This two-phase approach prevents the common problem of AI agents diving into implementation before understanding the architecture.

Session Persistence

OpenCode uses SQLite to persist session data across terminal sessions. This means you can close your terminal, come back later, and pick up where you left off.

What Gets Persisted

Conversation history (messages and responses)
File changes made during the session
Agent state (Plan vs. Build mode)
Active context (which files were being discussed)

Session Management

Start a new session for unrelated work
Continue an existing session when resuming previous work
Clear session history when accumulated context becomes counterproductive

Context Compaction

For long sessions, OpenCode supports context compaction. This summarizes older conversation history to free up context window space while retaining the essential information. Compaction is automatic and configurable: you can control how aggressively it summarizes based on your model's context window size.

This is particularly important when using models with smaller context windows (like local Ollama models with 8K or 32K contexts) where every token counts. Cloud models with 128K or 200K windows have much more room, but even they benefit from compaction during extended sessions.

Context Window Management Across Providers

Different providers offer different context window sizes, and your strategy should adapt:

Provider Tier	Context Size	Strategy
Small (8K-32K)	Ollama local models	Aggressive compaction, focused sessions, minimal background context
Medium (64K-128K)	GPT-4o, Claude Sonnet	Standard compaction, moderate session length, room for codebase context
Large (200K+)	Claude Opus, Gemini Pro	Minimal compaction needed, can handle long sessions with extensive context

Understanding your working model's context limit helps you decide how much context to load via opencode.json versus providing interactively. With a small local model, lean heavily on precise include patterns to keep only the most relevant files in context. With a large cloud model, you can afford broader context.

MCP Server Support

OpenCode supports MCP through the opencode mcp command, providing integration with external tools and data.

Configuration

# Add an MCP server
opencode mcp add my-db-server -- npx @my-org/db-mcp-server

# List configured servers
opencode mcp list

# Remove a server
opencode mcp remove my-db-server

MCP servers can also be configured in opencode.json:

{
  "mcp": {
    "servers": {
      "filesystem": {
        "command": "npx",
        "args": ["-y", "@anthropic/mcp-server-filesystem", "./"]
      }
    }
  }
}

When to Use MCP with OpenCode

The same principles apply as with other terminal agents: use MCP when the task requires data from outside the codebase (databases, APIs, external services). For code-only work, OpenCode's built-in file access is sufficient.

One consideration specific to OpenCode: if you are using a local model through Ollama, MCP adds server-side processing that runs locally. There is no additional privacy concern since everything stays on your machine.

LSP Integration: Real-Time Code Intelligence

OpenCode integrates with Language Server Protocol services to provide richer code context. LSP gives the agent:

Type information and function signatures
Import resolution and dependency tracking
Error and warning diagnostics from your language's toolchain
Symbol navigation and reference finding

This means OpenCode understands your code at a deeper level than simple text analysis. When you ask about a function, the agent knows its type signature, where it is called from, and what it depends on.

Why LSP Matters for Context

LSP provides structured context that would otherwise require the agent to infer from raw code. Knowing that a variable is of type List[UserModel] is more precise than the agent guessing from how the variable is used. This structured understanding reduces errors and produces more accurate code generation.

Custom Commands

OpenCode supports user-defined custom commands that encapsulate common operations with predefined context:

{
  "commands": {
    "review": {
      "description": "Review the current branch for issues",
      "prompt": "Review all changes in the current branch compared to main. Check for: security issues, performance problems, missing error handling, and test coverage gaps."
    },
    "test-all": {
      "description": "Run and analyze the full test suite",
      "prompt": "Run the complete test suite. Report any failures, flaky tests, or tests that take unusually long. Suggest fixes for any failures."
    }
  }
}

Custom commands combine a descriptive name with a predefined prompt, creating reusable context bundles for common workflows.

Thinking About Context Levels in OpenCode

Minimal Context

For quick questions about the codebase, just ask. OpenCode will explore files as needed.

Moderate Context

For feature work, set up your opencode.json with clear instructions and use the Plan agent first to establish understanding before switching to Build.

Heavy Context

For complex refactoring or architectural changes, combine: detailed opencode.json instructions, the Plan agent for architecture analysis, MCP servers for database or service context, and custom commands for verification steps.

External Documents: PDFs vs. Markdown

Markdown Is Preferred

OpenCode works with text-based formats. Project context documents, architecture decision records, and coding standards should be Markdown files in your repository.

PDFs

If you have reference material in PDF format, convert the relevant sections to Markdown. OpenCode does not have built-in PDF parsing, so text-based formats are more reliable.

Advanced Patterns

The Privacy-First Development Pattern

Use Ollama with a local model for sensitive codebases:

Install Ollama and download a capable model (Llama 3.1, Mistral Large, etc.)
Configure opencode.json to use the local Ollama endpoint
All context stays on your machine with zero network calls

This is particularly valuable for proprietary code, pre-launch features, or security-sensitive applications.

The Plan-Then-Build Pattern

Start with the Plan agent to analyze the codebase
Discuss the architecture and design approach
Switch to Build once you agree on the plan
Use custom commands to verify the implementation

The Multi-Provider Context Strategy

Use different providers for different context needs:

A large cloud model (GPT-4o, Claude Opus) for complex architectural planning
A fast, small model for quick edits and simple tasks
A local model for sensitive code that should not leave your machine

Switch providers in opencode.json based on the current task.

Common Mistakes

Not configuring opencode.json. Without it, OpenCode has no project context beyond what it can infer from file exploration.
Using Build when you should Plan. Jumping to code changes without planning leads to rework. Use the Plan agent first for anything non-trivial.
Ignoring context compaction. With smaller model context windows, long sessions degrade quality. Let compaction do its job, or start fresh sessions.
Not leveraging LSP. Ensure your language's LSP server is installed and running. The structured code intelligence significantly improves agent accuracy.
Skipping custom commands for repeated tasks. If you run the same kind of review or test analysis frequently, create a custom command.
Using cloud models for sensitive code without consideration. If code privacy matters, use Ollama with local models. The trade-off is sometimes reduced capability, but the privacy guarantee is absolute.

Go Deeper

To learn more about AI-assisted development and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Google Antigravity: A Complete Guide to the Agent-First IDE

Alex Merced — Sat, 07 Mar 2026 17:00:00 GMT

Google Antigravity is an agent-first IDE built by Google DeepMind's Advanced Agentic Coding team. It approaches context management differently from other AI coding tools because it is designed from the ground up around agentic workflows, where the AI is not just an assistant responding to prompts, but an autonomous agent that plans, executes, tracks progress, and retains knowledge across sessions. Its context management system centers on three pillars: Skills for reusable capability, Knowledge Items for persistent memory, and Artifacts for transparent documentation of its work.

This guide covers how to structure and manage context in Antigravity to get the most from its agentic capabilities.

How Antigravity Manages Context

Antigravity assembles its working context from multiple sources, layered by persistence:

Knowledge Items (KIs) - persistent, distilled knowledge from past conversations
Skills (SKILL.md files) - reusable instruction sets for specific capabilities
Workflows - step-by-step guides in the .agents/workflows/ directory
Conversation history - the current and past interactions
The codebase - files, directories, and project structure
MCP servers - external tools and data sources
Task artifacts - implementation plans, walkthroughs, and checklists the AI creates

What makes Antigravity distinctive is that it actively generates and maintains its own context artifacts. The AI creates task checklists, implementation plans, and walkthroughs as it works, and these become part of the persistent context for future sessions.

Skills: Reusable Capability Packages

Skills are Antigravity's primary mechanism for defining reusable capabilities. Each Skill is a folder containing a SKILL.md file with YAML frontmatter and detailed Markdown instructions.

Skill Structure

.agents/skills/
  my-skill/
    SKILL.md          # Required: instructions with YAML frontmatter
    scripts/          # Optional: helper scripts
    examples/         # Optional: reference implementations
    resources/        # Optional: templates or assets

SKILL.md Format

---
name: deploy-to-staging
description: Deploy the application to the staging environment
---

## Prerequisites
- Docker must be installed and running
- AWS CLI must be configured with staging credentials
- The current branch must have passing CI

## Steps
1. Build the Docker image with the staging configuration
2. Push the image to ECR
3. Update the ECS task definition
4. Trigger the deployment
5. Verify the health check endpoint responds

## Verification
- Check that the /health endpoint returns 200
- Verify the deployed version matches the expected Git SHA
- Run the smoke test suite against staging

When to Create Skills

Create a Skill when you have a workflow that:

You perform more than once
Requires specific steps in a specific order
Benefits from consistent execution across team members
Involves domain knowledge that is not obvious from the codebase alone

Skills vs. Other Context Mechanisms

Skills are for procedural knowledge ("how to do X"). They differ from:

Knowledge Items which store factual knowledge ("what is X")
GEMINI.md or CLAUDE.md style files which provide ambient project context
Artifacts which document specific work done in a specific session

Knowledge Items: Persistent Memory Across Conversations

Knowledge Items (KIs) are Antigravity's mechanism for retaining knowledge across conversations. Unlike conversation history (which is session-bound), KIs are distilled, curated facts that persist indefinitely.

How KIs Work

At the end of each conversation, a separate Knowledge Subagent analyzes the conversation and extracts key information into KIs. Each KI has:

metadata.json: summary, timestamps, references to original conversations
artifacts/: related files, documentation, and analysis

KIs are stored in the Knowledge directory and are automatically loaded when starting new conversations. Antigravity checks KI summaries at the beginning of every session to avoid redundant work.

What Gets Stored as KIs

Architecture decisions and their rationale
Troubleshooting discoveries and resolutions
Implementation patterns specific to your project
Configuration details and their implications
Integration specifics for external services
Performance characteristics and optimization strategies

Using KIs Effectively

The most important rule for KIs is: always check them before starting research. If you are about to analyze a codebase module, check whether a KI already covers that analysis. This prevents redundant work and ensures continuity across sessions.

You can also reference specific KIs in conversations by pointing Antigravity at the KI's artifact files. This is especially useful when building on previous work or when onboarding new team members who can review the accumulated KIs.

Artifacts: The Transparency System

Antigravity creates artifacts as structured Markdown documents that make the agent's work transparent and reviewable. Key artifact types include:

task.md

A checklist that tracks progress on the current task. Antigravity creates this at the start of complex work and updates it as it progresses:

# Feature: User Authentication

- [x] Research existing auth patterns
- [x] Create implementation plan
- [/] Implement JWT token generation
- [ ] Add refresh token support
- [ ] Write integration tests
- [ ] Update API documentation

implementation_plan.md

Created during the PLANNING phase, this documents the proposed changes, file modifications, and verification strategy before any code is written. You review and approve (or modify) this plan before Antigravity proceeds to execution.

walkthrough.md

Created after completing work, this documents what was accomplished, what was tested, and the results. It serves as a record of the work and can be reviewed by team members.

Why Artifacts Matter for Context

Artifacts create a structured record that Antigravity can reference in future sessions. When you return to a project, the agent can read the previous implementation plan and walkthrough to understand what was done and why. This is far more efficient than re-analyzing the codebase from scratch.

Thinking About Context Levels in Antigravity

Minimal Context (Quick Tasks)

For simple questions or small fixes, just ask. Antigravity can explore the codebase, read relevant files, and provide answers without additional setup. Its file exploration tools are fast and respect .gitignore patterns.

Moderate Context (Feature Work)

For typical feature development, let Antigravity's Planning phase do the heavy lifting. It will:

Analyze the codebase to understand the current architecture
Create an implementation plan for your review
Execute the plan once approved
Verify the changes

The PLANNING > EXECUTION > VERIFICATION workflow is built into Antigravity's DNA, and each phase generates artifacts that carry context forward.

Comprehensive Context (Ongoing Projects)

For sustained work across multiple sessions, invest in Skills and ensure KIs are accumulating properly. Over time, Antigravity builds a rich knowledge base about your project that makes each subsequent session more productive.

Multi-Model Support and Context Routing

Antigravity supports multiple AI models and can use different models for different subtasks. This means context management extends to model selection: some tasks benefit from larger context windows, while others benefit from faster inference.

The agent handles this transparently, but being aware of it helps you understand why some responses might take longer (larger model processing more context) while others are faster (smaller model handling a focused subtask).

Browser Recording and Visual Context

Antigravity includes a built-in browser interaction system that records all browser actions as WebP videos. This creates a unique form of context: visual proof of work that can be reviewed later.

For frontend development, this means Antigravity can:

Navigate to web applications and interact with UI elements
Take screenshots to verify visual changes
Record step-by-step interactions for documentation

These recordings become part of the walkthrough artifact, providing visual evidence that changes work as intended.

Conversation History and Context Summaries

Antigravity maintains conversation logs and summaries that persist across sessions. When you start a new conversation, the system provides:

Summaries of recent conversations
KI summaries with artifact paths
Information about previously edited and viewed files

This means Antigravity starts each session with awareness of what happened in recent sessions, reducing the need to re-explain context that was covered before.

MCP Server Support

Antigravity supports MCP servers for connecting to external tools and data sources. Configuration follows the standard MCP pattern familiar from other tools.

Practical Use Cases

Database access: Let Antigravity query your development database to understand schema and data
Browser automation: Verify frontend changes visually
Git hosting: Interact with GitHub or GitLab for PR management
Documentation systems: Access internal wikis or knowledge bases

When to Use MCP

Use MCP when the task requires information from outside the codebase. For code-only work, Antigravity's built-in file system tools are sufficient. MCP adds the most value for tasks that span multiple systems (for example, updating both code and documentation, or verifying a code change against a running application).

External Documents: PDFs vs. Markdown

Markdown Is the Native Format

Skills, KIs, and artifacts are all Markdown. If you are creating context documents for Antigravity, use Markdown.

For External References

PDF documents can be provided as context through conversation uploads. However, for persistent reference material, converting to Markdown and placing it in a project directory (or as a Skill resource) provides better integration with Antigravity's context system.

Advanced Patterns

The Skill-Driven Development Pattern

Create Skills for every major workflow in your development process:

deploy-staging for deployment
create-api-endpoint for new endpoints
database-migration for schema changes
security-audit for security reviews

When you need to perform one of these tasks, point Antigravity at the relevant Skill. This ensures consistent execution regardless of which team member is working.

The Knowledge Accumulation Pattern

Treat KIs as a growing knowledge base about your project:

First session: Antigravity learns the basic architecture
Subsequent sessions: KIs accumulate details about specific modules, patterns, and decisions
Over time: Antigravity starts with a deep understanding of your project every session

This compounds over weeks and months, making the AI increasingly effective.

The Paired Review Pattern

Use Antigravity's PLANNING phase as a design review:

Describe the feature or change you want
Review the implementation plan Antigravity creates
Provide feedback and iterate on the plan
Only approve execution once the plan meets your standards

This catches design issues before code is written, saving significant time.

The Task Decomposition Pattern

For large features, let Antigravity break the work into multiple task boundary segments:

Tell Antigravity the overall goal
It creates a task.md with subtasks
Each subtask gets its own PLANNING > EXECUTION > VERIFICATION cycle
The walkthrough artifact captures the full story for future reference

Common Mistakes

Ignoring KI summaries. Antigravity provides KI summaries at the start of each conversation. Skipping them leads to redundant work and missed context.
Not creating Skills for repeatable work. If you find yourself explaining the same workflow multiple times, it should be a Skill.
Skipping the PLANNING phase. Jumping straight to execution means no implementation plan to review. The PLANNING phase is where Antigravity aligns with your intent.
Not reviewing artifacts. Implementation plans and walkthroughs are designed for human review. Skipping them defeats the purpose of Antigravity's transparency system.
Over-relying on conversation context. Conversation history is ephemeral. For information that should persist, ensure it gets captured in Skills or KIs.
Not building Workflows for common tasks. The .agents/workflows/ directory supports step-by-step guides that Antigravity follows precisely. These are particularly useful for onboarding, deployment, and maintenance tasks.

Go Deeper

To learn more about working effectively with AI coding agents and managing context across development workflows, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Gemini CLI: A Complete Guide to Terminal-Native AI Development

Alex Merced — Sat, 07 Mar 2026 16:00:00 GMT

Gemini CLI is an open-source terminal agent powered by Gemini models that operates directly in your command line. It brings Google's AI capabilities into the environment where many developers already live, with a context management system built around hierarchical configuration files, persistent memory, MCP server integration, and direct codebase interaction. Unlike web-based tools where context is managed through uploads and conversation, Gemini CLI assembles its context from your project structure, your instruction files, and the tools you connect to it.

This guide covers every context management mechanism in Gemini CLI and explains how to configure them for productive development workflows.

How Gemini CLI Assembles Context

Gemini CLI builds its working context from multiple sources, loaded in a specific hierarchy:

Global GEMINI.md (~/.gemini/GEMINI.md) - personal preferences that apply everywhere
Project GEMINI.md (in your project directory, walking up to the root) - project conventions
Subdirectory GEMINI.md files - component-specific instructions
Memory entries - facts you have told the CLI to remember
MCP server tools - external data sources and capabilities
The current codebase - files, dependencies, project structure
The conversation - your prompts and responses in the current session

More specific sources take precedence over general ones. A subdirectory GEMINI.md instruction overrides a project-level GEMINI.md instruction on the same topic.

GEMINI.md: Persistent Project Context

GEMINI.md is the foundational context mechanism. It is a Markdown file that Gemini CLI loads automatically before every interaction.

File Hierarchy

Location	Scope	Purpose
`~/.gemini/GEMINI.md`	All projects	Personal coding style, universal preferences
`./GEMINI.md` (project root)	Current project	Architecture, stack, conventions
`./src/GEMINI.md`	Specific directory	Module-specific patterns

What to Include

# GEMINI.md

## Project: E-Commerce API
- Framework: Express.js on Node 22
- Database: PostgreSQL 16 with Drizzle ORM
- Testing: Vitest with supertest for API tests
- Deployment: Docker containers on Cloud Run

## Code Conventions
- Use ESM imports (no CommonJS require)
- All route handlers are async functions
- Error handling uses a centralized error middleware
- SQL migrations use Drizzle Kit

## Architecture
- Routes: src/routes/
- Services: src/services/ (business logic)
- Models: src/models/ (Drizzle schema)
- Middleware: src/middleware/
- Tests: tests/ (mirrors src/ structure)

## Do Not
- Do not use default exports
- Do not install packages without noting them
- Do not modify migration files after they have been applied

Modular GEMINI.md Files

For complex projects, GEMINI.md files can import other Markdown files. This keeps individual files focused while allowing the CLI to assemble comprehensive context:

# GEMINI.md
See also:
- @docs/coding-standards.md
- @docs/api-conventions.md
- @docs/testing-strategy.md

The /init Command

If you are starting a new project or onboarding Gemini CLI to an existing one, run /init. This command analyzes your project structure and generates a starting GEMINI.md file that captures:

Detected frameworks and languages
Project structure
Build and test commands
Basic conventions inferred from the code

Review and edit the generated file. The auto-detection is a starting point, not a finished product. Add your team's conventions, architectural decisions, and quality standards to make it comprehensive. The value of /init is that it saves you from writing the boilerplate sections (project type, folder structure, detected dependencies) so you can focus on the human-knowledge sections.

Memory: Persistent Facts Across Sessions

Gemini CLI's memory system stores persistent facts that apply across all sessions and projects (when stored globally).

Adding Memories

/memory add We use the Google Python Style Guide for all Python code
/memory add Our PostgreSQL database runs on port 5433, not the default 5432
/memory add Always use UTC timestamps in database columns

Viewing Memories

/memory show

This displays all active memories, including those from GEMINI.md files and explicit memory entries.

Refreshing Context

If you update GEMINI.md files outside of the current session, use:

/memory refresh

This reloads all context files without restarting the CLI.

Memory Best Practices

Use memory for facts that are true across projects (your personal conventions)
Use GEMINI.md for project-specific context
Keep memories concise: "Use Ruff for Python linting" rather than a paragraph explaining why
Review memories periodically with /memory show and remove outdated entries

Direct Context Injection with @

The @ command lets you inject specific files or directories directly into a prompt:

@src/models/user.ts How should I add a preferences field to this model?

@src/routes/ Review all route handlers for consistent error handling

This is the most direct way to give Gemini CLI context about specific files. Unlike other tools that require uploads, the @ command reads from your local file system in real time.

When to Use @

When your question relates to specific files that Gemini CLI might not automatically discover
When you want to ensure the agent reads the latest version of a file
When you want to focus the agent on a particular section of the codebase

MCP Server Support

Gemini CLI supports MCP through its settings.json configuration. MCP servers extend the CLI's capabilities by connecting it to external tools and data sources.

Configuration

MCP servers are configured in settings.json:

{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-server-github"]
    },
    "postgres": {
      "httpUrl": "http://localhost:3001/mcp"
    },
    "custom-tool": {
      "command": "python",
      "args": ["./scripts/my-mcp-server.py"],
      "env": {
        "API_KEY": "${MY_API_KEY}"
      }
    }
  }
}

Note the environment variable expansion (${MY_API_KEY}), which lets you keep credentials out of configuration files.

Transport Options

Gemini CLI supports three MCP transport mechanisms:

stdio: The server runs as a local process (most common for development)
SSE (Server-Sent Events): For remote servers using the url property
HTTP Streaming: For modern HTTP-based servers using the httpUrl property

MCP Prompts as Slash Commands

MCP servers can expose predefined prompts as slash commands. If a connected server exposes a prompt named "analyze-performance," you can invoke it with /analyze-performance directly in the CLI.

When to Use MCP

Use MCP for: Database access, GitHub integration, browser automation, accessing internal APIs, connecting to project management tools

Skip MCP when: The task is code-only and the files are already on your local system. Gemini CLI can read files and run terminal commands directly without MCP.

Dynamic Shell Context

One of Gemini CLI's unique strengths is its ability to execute shell commands to gather real-time context. This means the agent can check the actual state of your system rather than relying on static descriptions.

Practical Use Cases

Check current Git state: The agent can run git status or git log to understand what has changed recently, which branch you are on, and what commits are pending
Inspect running services: Commands like docker ps or kubectl get pods give the agent visibility into your running infrastructure
Read live configuration: The agent can check environment variables, read .env files, or inspect running process configurations
Verify test results: Running your test suite and analyzing the output gives the agent concrete data about what is passing and what is failing

This dynamic context is especially valuable for debugging workflows, where the agent needs to understand both the code and the runtime environment.

Automatic Codebase Exploration

Gemini CLI automatically explores your project structure using tools that respect .gitignore patterns. It will not waste context on node_modules/, __pycache__/, or build output. It also detects project types from configuration files (for example, finding package.json tells it this is a Node.js project).

This automatic exploration means you can ask broad questions like "What database does this project use?" and the agent will find the answer by scanning relevant configuration files. However, GEMINI.md files significantly improve results by providing context that cannot be inferred from code alone: team decisions, architectural rationale, and development philosophy.

Choosing Gemini CLI vs. Other Terminal Agents

Choose Gemini CLI over Claude Code when: You prefer Google's Gemini models, need the hierarchical GEMINI.md system, or want MCP prompts exposed as slash commands.

Choose Gemini CLI over OpenCode when: You want a simpler, more focused tool without OpenCode's TUI interface, or you are already invested in the Google ecosystem.

Choose Gemini CLI over Codex CLI when: You want an open-source tool you can inspect and modify, or you prefer interactive terminal sessions over Codex's sandbox model.

Thinking About the Right Level of Context

For Quick Questions

Just ask. Gemini CLI can explore your codebase on its own:

What database ORM does this project use?

The CLI will scan your project files, find the relevant configuration, and answer.

For Targeted Changes

Provide file references and constraints:

@src/services/auth.ts Add rate limiting to the login function. 
Use express-rate-limit with a 100-request-per-minute window.

For Large Features

Invest in GEMINI.md, set up relevant MCP servers, and use the multi-step approach: plan first, then implement.

External Documents: PDFs vs. Markdown

Markdown Is Native

GEMINI.md files, memory entries, and context documents should all be Markdown. The format is native to Gemini CLI's context system.

PDFs Need Conversion

Gemini CLI primarily works with text-based formats. If you have reference material in PDF form, extract the relevant sections into Markdown files and place them in your project directory. This makes them accessible via @ references and GEMINI.md imports.

Advanced Patterns

The Context-Aware Shell Script

Create shell scripts that set up project context before launching Gemini CLI:

#!/bin/bash
# Start Gemini CLI with project-specific context
cd ~/projects/my-api
export DB_URL="postgresql://dev@localhost:5433/mydb"
gemini

This ensures the CLI starts in the right directory with the right environment variables, reducing context-switching overhead.

The Exploration-First Pattern

Before starting a new feature:

Analyze the current authentication system.
Describe the flow from login to token validation.
Do not make any changes.

Review the analysis, correct any misunderstandings, and then proceed with the implementation task.

The Automated Context Generation Pattern

Use the /init command periodically (or a custom script) to regenerate your GEMINI.md file as the project evolves. This keeps the context file synchronized with the actual state of the codebase.

Common Mistakes

No GEMINI.md. Without it, the CLI starts with no project context. It can still explore your codebase, but it will make assumptions that may not match your conventions.
Stale GEMINI.md. A GEMINI.md that references frameworks or patterns you no longer use creates confusion. Update it when you make significant changes.
Overloading memory. Memory is for brief, stable facts. Do not try to store entire documents as memory entries.
Adding unnecessary MCP servers. Each connected server adds tools that the CLI must evaluate. Only connect servers you actively use.
Not using @ for targeted questions. Pointing Gemini CLI at specific files with @ produces more focused results than letting it search the entire project.
Ignoring /init. For new projects, /init generates a solid starting GEMINI.md in seconds. Review and refine it rather than writing from scratch.
Forgetting to refresh after external edits. If you edit GEMINI.md files in your text editor, run /memory refresh so the CLI picks up the changes immediately.
Writing overly long GEMINI.md files. GEMINI.md should be focused and scannable. If it exceeds 500 lines, consider splitting it into modular imported files. A concise GEMINI.md with clear sections is more effective than a sprawling document.

Go Deeper

To learn more about working effectively with AI coding agents and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Gemini Web and NotebookLM: A Complete Guide to Google's AI Knowledge Ecosystem

Alex Merced — Sat, 07 Mar 2026 15:00:00 GMT

Google's AI ecosystem for knowledge work consists of two deeply integrated tools: Gemini (the conversational AI at gemini.google.com) and NotebookLM (the research-focused assistant at notebooklm.google.com). In early 2026, these two platforms became interoperable, allowing Gemini to access information stored in NotebookLM notebooks. This integration creates something unique in the AI landscape: a persistent knowledge infrastructure where documents you upload once become available across both conversational and research interfaces.

This guide covers context management strategies for both Gemini Web and NotebookLM, with a focus on how to use them together for maximum effectiveness.

Gemini Web: Context Management Fundamentals

The Context Window

Gemini supports one of the largest context windows available, with models like Gemini 3 Pro and Gemini 2.5 Pro offering up to 2 million tokens. This is approximately 1.5 million words of input capacity, enough to process entire books, large codebases, or years of financial data in a single conversation.

The context window includes everything: your system instructions, conversation history, uploaded files, and Gemini's responses. While 2 million tokens is enormous, strategic context management still matters because relevance, not volume, determines response quality.

Custom Instructions

Gemini supports custom instructions that shape how it responds across conversations. Access these through Gemini's settings. Effective custom instructions include:

Your professional background and expertise level
Preferred response style (concise vs. detailed, formal vs. conversational)
Output format preferences (bullet points, structured sections, code formatting)
Domain-specific terminology or constraints

Gems: Specialized AI Assistants

Gems are custom AI mini-apps within Gemini. You create a Gem by defining its purpose, instructions, and behavior. Unlike custom instructions (which apply globally), each Gem operates with its own specialized context.

Use Gems for repeatable workflows:

A "Technical Writer" Gem with your style guide and terminology baked in
A "Data Analyst" Gem that knows your preferred visualization tools and analysis frameworks
A "Meeting Prep" Gem that generates agendas and briefing documents in your format
A "Code Reviewer" Gem that applies your team's coding standards consistently
A "Content Editor" Gem that checks for brand voice compliance

To create a Gem, navigate to the Gems section in Gemini, define its instructions, and optionally upload knowledge files. Once created, you can invoke the Gem anytime without re-establishing its context.

Building Effective Gems

The quality of a Gem depends entirely on the quality of its instructions. Write Gem instructions as if you are onboarding a new team member to a specific role:

Define the Gem's role ("You are a technical documentation editor for a developer tools company")
Specify the audience ("Write for senior developers who know the basics but need guidance on advanced topics")
Set quality standards ("Every section must include at least one code example, use active voice, and stay under 300 words per subsection")
Include anti-patterns ("Never use jargon without defining it first, never assume the reader has used this tool before")
Provide examples of the desired output style when possible

Notebooks (Projects)

Gemini is rolling out "Notebooks" (an evolution of its Projects feature) that let you group conversations by topic and set per-notebook custom instructions. This mirrors the Project concept in other AI tools: a workspace where context persists across conversations.

Within a Notebook:

Set instructions specific to the topic or project
Upload files that Gemini can reference in every conversation
Maintain a collection of related conversations without losing context between them

File Uploads

Gemini Web supports direct file uploads in conversations:

File Type	Use Case
PDF	Research papers, specifications, reports
Documents	Google Docs, Word files for editing or analysis
Spreadsheets	Data analysis, financial modeling
Images	Visual context, screenshots, diagrams
Audio	Transcription and analysis
Video	Visual content analysis

Google Workspace Integration

A distinctive Gemini feature is its integration with Google Workspace. With "Personal Intelligence" (available in 2026), Gemini can securely access your Gmail, Drive, Docs, and Calendar to provide context-aware responses grounded in your actual work data. This means Gemini can:

Search your email history for relevant communications
Reference documents in your Google Drive
Check your calendar when you ask about scheduling
Pull data from your spreadsheets for analysis

This integration effectively makes your entire Google Workspace a context source, something no other AI platform currently matches.

NotebookLM: Deep Research Context Management

NotebookLM is purpose-built for research and knowledge work. Its context management is centered around "notebooks," each of which contains sources (your uploaded documents) and a conversation interface grounded in those sources.

How NotebookLM Handles Context

Unlike Gemini (which can draw on its entire training data), NotebookLM responses are grounded exclusively in the sources you upload. This is a feature, not a limitation. When you need answers based specifically on your documents (not the model's general knowledge), NotebookLM provides citation-backed responses that reference specific sections of your sources.

Source Types

NotebookLM supports a wide range of source types:

PDFs: Research papers, reports, legal documents
Google Docs: Your own writing, notes, and drafts
Google Slides: Presentation content
Web URLs: Articles, documentation, and reference pages
YouTube videos: Automatic transcription and analysis
Audio files: Podcast episodes, interviews, lectures
Text files: Any plaintext content

Free tier: Up to 50 sources per notebook (500,000 words or 200MB per source) NotebookLM Pro: Up to 300 sources per notebook

Custom Instructions in NotebookLM

NotebookLM supports per-notebook custom instructions. You can set:

Response style ("Explain like I am new to this field")
Response length preferences
Tone (academic, conversational, technical)
Specific focus areas within your sources

Audio Overviews

NotebookLM's Audio Overview feature generates podcast-style discussions of your uploaded sources. This is a unique context consumption approach: instead of reading AI-generated summaries, you listen to a natural conversation about your documents. Audio Overviews are useful for:

Getting a high-level understanding of dense material before deep-reading
Reviewing research while multitasking
Sharing knowledge with colleagues who prefer audio formats

Using Gemini and NotebookLM Together

The integration between Gemini and NotebookLM is where the real power emerges.

The Knowledge Flow

Upload sources to NotebookLM: Research papers, reports, specifications
Let NotebookLM build a grounded knowledge base: Ask questions, generate summaries, create Audio Overviews
Import that notebook into Gemini: Gemini gains access to all your NotebookLM sources
Use Gemini for broader analysis: Gemini combines your specific sources with its general knowledge and web search

This workflow gives you both grounded, citation-backed analysis (NotebookLM) and broader contextual understanding (Gemini) from the same source material.

When to Use Each

Need	Use
Answers grounded strictly in your documents	NotebookLM
Broad research with web search integration	Gemini Web
Citation-backed analysis of specific papers	NotebookLM
Creative ideation and brainstorming	Gemini Web
Audio summaries of research material	NotebookLM
Integration with Google Workspace data	Gemini Web
Side-by-side comparison of source documents	NotebookLM

Gems as Auto-Syncing Brains

When you create a Gem that is linked to a NotebookLM notebook, the Gem automatically stays in sync with the notebook's sources. Add a new document to the notebook, and the Gem can reference it immediately. This creates a "specialized brain" that continuously learns from your latest research without requiring you to re-upload files or restate context.

External Documents: PDFs vs. Markdown

In NotebookLM

NotebookLM works well with PDFs because it extracts and indexes the content for citation-backed retrieval. Since NotebookLM's primary job is to ground responses in specific documents, PDFs are perfectly suited for this use case.

However, for your own notes, outlines, and structured reference material, Google Docs or Markdown files (uploaded as text) provide cleaner parsing and are easier to update.

In Gemini Web

Gemini handles both PDFs and text-based formats well, but the same general rule applies: Markdown and plaintext provide the cleanest AI-parseable context. Use PDFs for published documents you received from others, and Markdown or Google Docs for context you author yourself.

MCP Server Support

As of early 2026, Gemini Web and NotebookLM do not support MCP (Model Context Protocol) server connections. MCP support is available in the Gemini CLI, which is covered in a separate guide.

For web-based Gemini usage, the Google Workspace integration provides similar benefits to MCP for many use cases: live access to your email, documents, spreadsheets, and calendar. If you need connections to non-Google services (databases, third-party APIs), use the Gemini CLI instead.

Advanced Patterns

The Research Pipeline

Collect sources in NotebookLM (upload papers, articles, reports)
Generate an Audio Overview for high-level understanding
Ask targeted questions in NotebookLM for citation-backed answers
Import the notebook to Gemini for broader analysis
Use Gemini with web search to find related work not in your sources
Draft your output in Gemini using both grounded sources and general knowledge

The Knowledge Base Strategy

Use NotebookLM notebooks as persistent knowledge bases for different domains:

"Industry Research" notebook with market reports and analyst papers
"Technical Reference" notebook with API docs and architecture papers
"Competitive Intelligence" notebook with competitor materials

Each notebook becomes a specialized resource that you can query independently or combine with Gemini for cross-domain analysis.

The Document Synthesis Pattern

When you need to synthesize multiple long documents:

Upload all documents to a single NotebookLM notebook
Ask NotebookLM to summarize each document individually
Ask it to identify common themes across all documents
Ask it to highlight contradictions or disagreements between documents
Use the results in Gemini for a final synthesized analysis

This approach leverages NotebookLM's grounding capability for accurate summarization and Gemini's broader intelligence for synthesis.

Structuring Context for Gemini and NotebookLM

In Gemini: Lead with Purpose

Because Gemini has such a large context window, it is tempting to dump everything in and hope for the best. Resist this. Structure your inputs:

State your goal first ("I need a comparison table of three database solutions")
Provide the relevant data (paste or reference uploaded files)
Specify the output format ("Create a markdown table with columns for Feature, Solution A, Solution B, Solution C")

This pattern works because Gemini prioritizes recent and explicit instructions over ambient context.

In NotebookLM: Trust the Grounding

NotebookLM is designed to answer from your sources. You do not need to paste content into the chat because the sources are already indexed. Instead, ask specific questions that require the AI to synthesize across your documents:

"Compare how Document A and Document B define the term 'data mesh'"
"What evidence in my sources supports the claim that real-time processing reduces costs?"
"Identify contradictions between the 2024 and 2025 reports on this topic"

Common Mistakes

Using Gemini when you need citations. If you need responses backed by specific sources, use NotebookLM. Gemini's general knowledge is powerful but cannot provide page-level citations from your documents.
Overloading a single NotebookLM notebook. While Pro supports 300 sources, having too many unrelated documents in one notebook dilutes the AI's focus. Create separate notebooks for distinct topics.
Not using Audio Overviews. Audio Overviews are one of NotebookLM's most underused features. They provide an efficient way to internalize complex material, especially before you start asking detailed questions.
Ignoring the Gemini-NotebookLM integration. Using these tools in isolation means you miss the most powerful workflow: grounded research in NotebookLM feeding into broader analysis in Gemini.
Skipping custom instructions. Both Gemini and NotebookLM support per-workspace custom instructions. Setting these up takes minutes and saves hours of course-correcting the AI.
Not using Gems for repeatable tasks. If you find yourself giving Gemini the same instructions repeatedly, create a Gem and save that context permanently.

Go Deeper

To learn more about context management strategies for AI tools, research workflows, and agentic systems, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Claude Code: A Complete Guide for Developers

Alex Merced — Sat, 07 Mar 2026 14:00:00 GMT

Claude Code is a terminal-native agentic coding assistant that lives in your command line and operates directly on your codebase. Unlike chat-based interfaces where you copy and paste code snippets, Claude Code reads your files, explores your project structure, runs commands, executes tests, and commits changes. Context management in Claude Code is about configuring the agent's persistent knowledge of your project so it can operate effectively without constant direction.

This guide covers every context management mechanism in Claude Code, from the foundational CLAUDE.md file to MCP integrations and multi-agent orchestration.

How Claude Code Manages Context

Claude Code builds its context from several sources, layered from most persistent to most ephemeral:

CLAUDE.md files (permanent project instructions)
MEMORY.md (automatically maintained session memory)
MCP server connections (live external data)
The codebase itself (files, dependencies, project structure)
The current conversation (your commands and the agent's responses)
Command output (terminal results, test output, error messages)

The agent combines all of these into a working context that informs how it approaches tasks. The most effective Claude Code users invest time in the persistent layers (CLAUDE.md, MEMORY.md, MCP) so that every conversation starts with a solid foundation.

CLAUDE.md: Your Project's Instruction Manual

CLAUDE.md is the primary mechanism for giving Claude Code persistent context about your project. It is a Markdown file that Claude reads at the start of every session.

File Locations and Hierarchy

Claude Code loads CLAUDE.md files from multiple locations, combining them into a single instruction set:

Location	Scope	Use For
`~/.claude/CLAUDE.md`	Global (all projects)	Personal preferences, universal standards
`./CLAUDE.md` (project root)	Project-wide	Architecture, coding standards, testing strategy
`./src/CLAUDE.md` (subdirectory)	Component-specific	Module-specific patterns, API conventions

More specific files supplement more general ones. If your global CLAUDE.md says "use 2-space indentation" but your project CLAUDE.md says "use 4-space indentation," the project-level instruction takes precedence.

What to Include in CLAUDE.md

# CLAUDE.md

## Project Overview
This is a Python FastAPI application with a React frontend.
Backend: Python 3.12, FastAPI, SQLAlchemy, PostgreSQL 16
Frontend: TypeScript, React 19, Vite 6, Zustand
Testing: pytest (backend), Vitest (frontend)

## Build and Run Commands
- Backend: `uvicorn app.main:app --reload`
- Frontend: `npm run dev`
- Tests: `pytest` (backend), `npm test` (frontend)
- Lint: `ruff check .` (backend), `npm run lint` (frontend)

## Code Conventions
- Use type hints for all function signatures
- Use Pydantic models for API request/response schemas
- Use async functions for all database operations
- Prefer composition over inheritance
- Keep functions under 30 lines; extract helpers for longer logic

## Testing Requirements
- Every new endpoint needs integration tests
- Every utility function needs unit tests
- Mock external services; never hit real APIs in tests
- Use factories (not fixtures) for test data creation

## Architecture Decisions
- We use the repository pattern for database access
- All business logic lives in the service layer, not in route handlers
- Frontend state is managed exclusively through Zustand stores
- API responses follow the JSON:API specification

CLAUDE.md Best Practices

Be specific and actionable. "Write clean code" is useless. "Functions should have a single responsibility and no side effects" is useful.
Include build and test commands. Claude Code will run these commands to verify its work. If it does not know your test command, it cannot validate changes.
Document your architecture. Tell Claude Code where things live. "Database models are in app/models/" saves the agent from exploring your entire project structure.
Use negative constraints. "Do not use class-based views" and "Never import directly from internal modules; use the public API" prevent common mistakes.
Keep it current. An outdated CLAUDE.md with references to deprecated patterns causes more harm than having no CLAUDE.md at all.

MEMORY.md: Automatic Session Memory

MEMORY.md is a file that Claude Code creates and maintains automatically to persist important context across sessions. When you share information that Claude determines is worth remembering (project decisions, your preferences, issue resolutions), it writes that information to MEMORY.md.

How MEMORY.md Works

Claude Code creates ~/.claude/MEMORY.md automatically
During conversations, when you share important context, Claude offers to save it
In subsequent sessions, Claude reads MEMORY.md before starting work
You can also manually edit MEMORY.md to add or remove memories

What Gets Stored

Typical MEMORY.md entries include:

Project preferences you have stated ("I prefer named exports over default exports")
Decisions you have made ("We chose Redis for session storage because of its TTL support")
Debugging discoveries ("The auth middleware requires the Authorization header in lowercase")
Workflow notes ("Always run migrations before testing database changes")

Managing MEMORY.md

Review MEMORY.md periodically. Like any persistent context, stale entries can lead the agent astray. Remove entries that no longer apply and update ones that have changed.

You can also use the /memory slash command during a session to view what Claude currently remembers.

Slash Commands: Real-Time Context Control

Claude Code provides several slash commands for managing context during a session:

Command	Purpose
`/context`	Show all active context sources
`/clear`	Clear conversation history (keeps CLAUDE.md and MEMORY.md)
`/agent`	Spawn a sub-agent for a specific task
`/memory`	View and manage session memories
`/help`	List available commands

Using /clear Strategically

Long sessions accumulate irrelevant context that can degrade Claude Code's focus. Use /clear when:

You are switching to a different part of the codebase
The conversation has gotten long and the agent seems confused
You want to start a focused task without the baggage of previous exchanges

Note that /clear preserves your CLAUDE.md and MEMORY.md context. Only the conversation history is reset.

Using /agent for Sub-Tasks

The /agent command spawns a sub-agent that operates independently with its own context. This is useful for:

Exploring a part of the codebase without polluting your main conversation
Running a time-consuming task (like a full test suite analysis) in parallel
Dividing a large feature into independent pieces

MCP Server Support

Claude Code supports MCP through the claude mcp command, allowing you to connect external tools and data sources.

Configuration

# Add a database MCP server
claude mcp add postgres -- npx @anthropic/mcp-server-postgres

# Add a filesystem MCP server
claude mcp add files -- npx @anthropic/mcp-server-filesystem /path/to/project

# List active MCP servers
claude mcp list

# Remove an MCP server
claude mcp remove postgres

Practical MCP Use Cases for Developers

Development database: Let Claude Code query your dev database to understand schema, check data state, and verify migrations.

Browser testing: Connect a Playwright MCP server so Claude Code can verify frontend changes by interacting with a running application.

Git hosting: Connect a GitHub or GitLab MCP server for creating pull requests, checking CI status, and reviewing code.

Documentation systems: Access internal docs or wikis that provide context not in the codebase.

When to Use MCP vs. Direct Commands

Claude Code can already run terminal commands. If you just need to see git log or psql -c "SELECT * FROM users", Claude Code can run those directly. MCP is more useful when:

The interaction is structured and repeatable (not ad-hoc commands)
You want Claude to have persistent access to a service across the entire session
The MCP server provides tools that are safer or more convenient than raw commands

External Documents: When to Use PDFs vs. Markdown

For Codebase Context: Always Markdown

CLAUDE.md, MEMORY.md, and any reference documents you create for Claude Code should be Markdown. The format is native to Claude Code's context system, version-controllable, and parses without ambiguity.

For External Specifications: Convert When Possible

If you have API specifications, design documents, or architecture diagrams in PDF form, consider extracting the relevant sections into Markdown and placing them in your repository. This way Claude Code can access them through normal file reading rather than requiring file upload.

For One-Off References

If you need Claude Code to reference a specific document during a session, paste the relevant content directly into the conversation. Claude Code's context window is large enough to handle substantial text inclusions.

Advanced Patterns

The Test-Driven Context Pattern

Write failing tests that describe the behavior you want
Tell Claude Code: "Make these tests pass"
The tests themselves become the context for the implementation

This is one of the most effective strategies because tests are unambiguous specifications. Claude Code does not need to interpret your prose when it has concrete pass/fail criteria.

The Progressive Codebase Understanding Pattern

When onboarding Claude Code to a new project:

Start with CLAUDE.md covering the basics (stack, structure, commands)
Ask Claude to explore the codebase and describe what it finds
Correct any misunderstandings and add clarifications to CLAUDE.md
Gradually delegate more complex tasks as the context matures

This iterative approach builds a robust CLAUDE.md faster than trying to write everything from scratch.

The Multi-Agent Feature Pattern

For large features with independent components:

Use /agent to spawn a sub-agent for each component
Main agent: coordinates the overall architecture
Sub-agent 1: implements the database layer
Sub-agent 2: implements the API endpoints
Sub-agent 3: implements the frontend components
Main agent: integrates the results and runs full tests

Each sub-agent operates with focused context, producing better results than one agent trying to build everything sequentially.

The Code Review Pattern

Use Claude Code as a reviewer before submitting your own PRs:

"Review the changes in the current branch compared to main. Check for: security issues, performance problems, missing error handling, test coverage gaps, and style guide violations from CLAUDE.md."

The persistent CLAUDE.md context means the review applies your project's specific standards, not generic best practices.

When to Choose Claude Code Over Other Tools

Choose Claude Code over Claude Web/Desktop when: Your task is code-centric and benefits from direct file system access, terminal command execution, and test running.

Choose Claude Code over OpenAI Codex when: You prefer a terminal-native interactive workflow over Codex's sandbox-and-PR model, or your project uses the Claude model family.

Choose Claude Code over Cursor or Windsurf when: You want a lightweight terminal agent without the overhead of a full IDE, or you work primarily in the terminal.

Common Mistakes

No CLAUDE.md. Claude Code still works without one, but it will make assumptions about your project that may not match reality. Ten minutes spent writing CLAUDE.md saves hours of corrections.
Stale CLAUDE.md. A CLAUDE.md that references a framework you migrated away from six months ago actively misleads the agent. Keep it current.
Not using /clear. Long sessions accumulate noise. Clear the conversation when switching tasks or when the agent seems to be losing focus.
Over-relying on MCP. If Claude Code can accomplish a task through direct file access and terminal commands, adding an MCP server is unnecessary overhead.
Ignoring MEMORY.md. Review it periodically. Claude Code's auto-generated memories are usually accurate, but occasionally they capture outdated or incorrect information.
Micro-managing the agent. Claude Code is designed for autonomous task execution. Give it a clear objective, ensure the context is correct, and let it work. Interrupting with constant corrections breaks the agent's flow.

Go Deeper

To learn more about working effectively with AI coding agents and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Claude CoWork: A Complete Guide for Knowledge Workers

Alex Merced — Sat, 07 Mar 2026 13:00:00 GMT

Claude CoWork represents a fundamentally different approach to AI context management. Unlike chat interfaces where you send messages and receive responses, CoWork is an autonomous agent that works on your local machine, reads and writes files directly, and executes multi-step tasks with minimal supervision. For knowledge workers who spend their days in documents, spreadsheets, and presentations, CoWork replaces the constant back-and-forth of copy-paste workflows with direct delegation.

This guide covers how to manage context effectively in CoWork, from setting up folder-level instructions to creating reusable workflows that run on schedule.

How CoWork Differs from Other Claude Interfaces

CoWork runs as part of the Claude Desktop application but operates in a distinct mode. The differences matter for context management:

Capability	Claude Web/Desktop Chat	Claude CoWork
Interaction model	Conversational (you send, it responds)	Autonomous (you delegate, it executes)
File access	Upload or MCP server	Direct local read/write
Output location	In the chat window	On your file system
Task duration	Minutes (conversational)	Minutes to hours (autonomous)
Scheduling	Manual only	Scheduled or on-demand
Sub-agents	No	Yes (parallel task decomposition)

Because CoWork operates autonomously on your local files, context management is less about what you say in a conversation and more about how you structure your file system, instructions, and task definitions.

Thinking About Context for Autonomous Tasks

When delegating to CoWork, the context equation changes. In a chat, you can course-correct in real time. With CoWork, you define the context upfront and the agent executes on its own. This means your context needs to be more complete and more explicit than in conversational interfaces.

Before Delegating, Ask:

Does the task have a clear, verifiable end state? "Organize these files by date" is clear. "Make these files better" is not.
Can I describe the success criteria in writing? If you cannot articulate what "done" looks like, CoWork will struggle too.
Does CoWork have access to everything it needs? Files, folders, reference material, and formatting instructions should all be accessible before you start.

The Delegation Spectrum

Simple delegation (minimal context): "Create a summary of every PDF in the /reports folder and save it as summary.md"

Moderate delegation: "Generate a weekly status report using the data in /projects/metrics.csv. Follow the format in /templates/weekly-report.md. Save the output to /reports/week-12-report.md"

Complex delegation: "Research the competitive landscape for product X by reading the documents in /research/competitors/. Create a presentation in PowerPoint format that covers market positioning, pricing comparison, and feature gaps. Use the company brand guidelines in /brand/style-guide.pdf for formatting."

Each level requires progressively more context, but all of it is provided through file access and instructions rather than conversation.

Global and Folder Instructions

CoWork uses a layered instruction system that lets you set context at different scopes.

Global Instructions

Global instructions apply across all CoWork tasks regardless of which folder or project you are working in. Set these for:

Your preferred writing style and tone
Output format preferences (bullet points vs. prose, heading structure)
General constraints ("Always use metric units," "Write in American English")
Your role and expertise level

These function similarly to Custom Instructions in ChatGPT but are specific to CoWork's autonomous execution mode.

Folder Instructions

Folder-level instructions apply when CoWork operates within a specific directory. This is where context management gets powerful. You can create different instruction sets for different projects:

/work/project-alpha/ might have instructions about project-specific terminology and formatting
/work/blog-drafts/ might have instructions about your blog style guide and target audience
/work/financial-reports/ might have instructions about compliance requirements and number formatting

Folder instructions override global instructions when they conflict, giving you precise control over CoWork's behavior in each context.

Writing Effective Instructions

Focus on what CoWork needs to know to complete tasks autonomously:

## Project Context
This folder contains marketing materials for Product X.
Target audience: enterprise IT decision-makers.
Tone: professional, authoritative, not salesy.

## File Organization
- /drafts/ contains work-in-progress documents
- /final/ contains approved, publication-ready content
- /assets/ contains images, charts, and data files
- /templates/ contains formatting templates

## Quality Standards
- All claims must be supported by data from the /assets/ folder
- Final documents must follow the template in /templates/standard.docx
- Run a readability check: target Flesch-Kincaid grade 10-12

MCP Server Integration

CoWork supports MCP (Model Context Protocol) through the Claude Desktop application's MCP configuration. MCP servers expand what CoWork can access beyond the local file system.

Useful MCP Servers for Knowledge Workers

Google Drive or OneDrive: Access cloud-stored documents without downloading them first

Notion or Confluence: Read from and write to your team's knowledge base

Slack: Pull conversation context or post updates about completed tasks

Calendar: Check scheduling context when preparing meeting materials

Email: Draft responses based on incoming email content

When MCP Adds Value for CoWork

MCP is most valuable when CoWork needs information from systems outside your local file system. If you are creating a report that combines local data with information from your company wiki, an MCP server for that wiki lets CoWork access both sources in a single task.

However, for purely local tasks (organizing files, generating documents from local data, processing spreadsheets), MCP adds unnecessary complexity. If the data is already on your machine, direct file access is simpler and faster.

Scheduled Tasks: Context That Runs Automatically

One of CoWork's distinctive features is task scheduling. You can define tasks that run at specific intervals (daily, weekly, monthly), and CoWork executes them with the same context every time.

Use Cases for Scheduled Tasks

Weekly report generation: Compile data from multiple sources into a formatted report every Monday
Daily email drafts: Prepare responses to routine communications based on templates
Monthly file organization: Sort and archive documents that have accumulated in download or inbox folders
Data processing: Transform incoming CSV exports into formatted spreadsheets at regular intervals

Context for Scheduled Tasks

Scheduled tasks need to be fully self-contained. The context must include:

Where to find inputs (file paths, folders to scan)
What to do with them (the processing logic)
Where to put outputs (destination paths)
What quality checks to apply (validation rules)
What to do when something unexpected happens (error handling)

Because you are not present during execution, the instructions must anticipate edge cases. For example: "If no new files are found in /inbox/, skip processing and do not create an empty report."

Sub-Agent Delegation

CoWork can decompose complex tasks into subtasks and execute them in parallel using sub-agents. This is particularly useful for tasks that involve independent workstreams.

How Sub-Agents Improve Context Management

Instead of providing one massive context for a complex task, CoWork breaks it into smaller, focused contexts:

Sub-agent 1: "Summarize the financial data in /data/q3-financials.csv"
Sub-agent 2: "Extract key quotes from the customer interviews in /research/interviews/"
Sub-agent 3: "Create a chart comparing year-over-year growth using the data in /data/growth.csv"

Each sub-agent gets a focused context, which typically produces better results than one agent trying to handle everything.

Monitoring Sub-Agent Progress

CoWork surfaces its reasoning and progress as it works. You can observe the plan, see which sub-agents are active, and intervene if something goes off track. This transparency is a context management feature itself because it lets you assess whether the agent's understanding matches your intent before it completes the task.

Working with External Documents

PDFs

CoWork can read PDFs directly from your file system. Use PDFs for:

Published specifications and standards
Research papers and reports from external sources
Contracts, legal documents, or compliance materials
Documents you received from others in PDF format

Markdown Files

CoWork excels with Markdown because the structure is unambiguous. Use Markdown for:

Your own notes, outlines, and instructions
Style guides and formatting templates
Context documents you create specifically for CoWork
Any document you plan to update frequently

The Hybrid Strategy

Keep critical reference material as Markdown in well-organized project folders. Use PDFs for external documents you cannot control. This gives CoWork the cleanest possible context for the documents you author and reasonable access to everything else.

Advanced Patterns

The Template-Driven Workflow

Create a template folder with examples of your desired output format. In your folder instructions, reference these templates. CoWork will pattern-match against them when generating new content.

/project/
  /templates/
    blog-post-template.md
    report-template.md
    email-template.md
  /instructions.md (folder instructions referencing templates)
  /output/

This approach gives CoWork concrete examples of "what good looks like" for every type of output it might produce.

The Progressive Delegation Pattern

Start with simple tasks to build confidence in CoWork's understanding of your context:

Week 1: File organization and simple summaries
Week 2: Document generation from templates
Week 3: Multi-source research and synthesis
Week 4: Complex deliverables with scheduled execution

Each phase lets you refine your instructions based on how CoWork interprets them.

The Quality Gate Pattern

For high-stakes outputs, set up a two-stage workflow:

Stage 1: CoWork generates a draft and saves it to /drafts/
Stage 2: You review the draft and provide feedback
Stage 3: CoWork revises based on your feedback and saves to /final/

This pattern combines autonomous execution with human review, giving you the efficiency of delegation without sacrificing quality control.

When to Use CoWork vs. Other Claude Interfaces

CoWork is not always the right choice. Here is how it compares for different scenarios:

Use CoWork when:

The task involves creating, transforming, or organizing files on your local machine
The work can be defined upfront with clear success criteria
You want to delegate entirely and come back to a finished result
The task is repeatable and benefits from scheduling

Use Claude Web when:

You want an interactive conversation to explore ideas or get feedback
The task is primarily knowledge-based (brainstorming, research questions, analysis)
You need artifacts like code demos or documents that persist in a conversation

Use Claude Desktop chat when:

You need MCP access to external services during an interactive conversation
You want Computer Use to interact with desktop applications
You need the conversational interaction model with live external data

Use Claude Code when:

You are working on a software codebase
You need the agent to navigate code, run tests, and make pull requests
You want terminal-level interaction with coding-specific tools

Common Mistakes

Vague task definitions. "Make these documents better" gives CoWork nothing to work with. Specify what "better" means: more concise, better formatted, restructured for a different audience, updated with new data.
Skipping folder instructions. Without instructions, CoWork uses only global context and its general training. Folder instructions are what make CoWork effective for your specific workflow.
Over-scoping tasks. A single task that says "create an entire marketing strategy" is too broad. Break it into research, analysis, drafting, and formatting phases.
Not reviewing outputs. CoWork runs autonomously, but that does not mean blindly accepting its output. Always review, especially for scheduled tasks that run without your active oversight.
Ignoring the file system. CoWork works with files. If your files are disorganized, CoWork's output will be disorganized. Invest in clean folder structures before delegating.
Underusing sub-agents. If a task has independent workstreams, let CoWork decompose it. Trying to force everything into a single linear execution path is slower and produces worse results.

Go Deeper

To learn more about working effectively with AI agents and context management strategies, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Claude Desktop: A Complete Guide to MCP, Computer Use, and Local File Access

Alex Merced — Sat, 07 Mar 2026 12:00:00 GMT

Claude Desktop takes everything available in Claude Web and adds three capabilities that fundamentally change how you manage context: MCP server connections that link Claude to external tools and data sources, direct local file access that eliminates the upload-download cycle, and Computer Use that lets Claude interact with your desktop environment. These additions make Claude Desktop the right choice when your work requires live data, local file system access, or integration with tools that Claude Web cannot reach.

This guide explains how to leverage each of Claude Desktop's context management features, when to use them, and how they complement the Projects, artifacts, and conversation patterns covered in the Claude Web guide.

What Claude Desktop Adds Over Claude Web

Claude Desktop shares the same core features as Claude Web: Projects with instructions and knowledge files, artifacts, and the same large context windows (up to 1 million tokens). The key additions are:

Feature	Claude Web	Claude Desktop
Projects	Yes	Yes
Artifacts	Yes	Yes
Knowledge files	Yes	Yes
MCP servers	No	Yes
Local file access	Upload only	Direct read/write
Computer Use	No	Yes (beta)

If your work is purely knowledge-based (writing, research, analysis), Claude Web is sufficient. Switch to Claude Desktop when you need to connect Claude to your local environment or external services.

MCP Servers: The Core Differentiator

The Model Context Protocol (MCP) is what makes Claude Desktop a genuinely different tool from the web interface. MCP is an open standard that allows Claude to connect to external services, databases, file systems, and tools through standardized server implementations.

How MCP Works in Claude Desktop

Claude Desktop acts as the MCP host. You configure MCP servers in the application settings, and Claude gains access to the tools those servers expose. When Claude needs information from an external source, it calls the appropriate MCP tool, receives the results, and incorporates them into its response.

Practical MCP Use Cases

Database Access: Connect a database MCP server to let Claude query your development database directly. Instead of copying and pasting query results, Claude can run queries itself:

Explore schema to understand your data model
Run diagnostic queries when debugging
Verify data after explaining a migration plan

File System Access: Connect a filesystem MCP server to give Claude access to specific directories on your machine. This is especially useful for:

Browsing project directories without manually uploading each file
Reading configuration files, logs, or data files
Writing output files (reports, generated code, processed data) directly to disk

Version Control: Connect a Git MCP server to let Claude interact with your repository:

Review recent commits and diffs
Understand the project's change history
Create branches or commits (with your approval)

API Integration: Connect MCP servers for services your workflow depends on:

Jira or Linear for project management context
Notion or Confluence for internal documentation
Slack for team communication context

Setting Up MCP Servers

MCP servers are configured in Claude Desktop's settings as JSON:

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-server-filesystem", "/path/to/project"]
    },
    "postgres": {
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-server-postgres"],
      "env": {
        "DATABASE_URL": "postgresql://user:pass@localhost:5432/mydb"
      }
    }
  }
}

When to Use MCP

Use MCP when:

Your task requires data that is not (and should not be) in the conversation or project files
You need Claude to interact with live systems (databases, APIs, file systems)
You want Claude to verify its work against real systems
The data changes frequently and uploading snapshots is impractical

Do not use MCP when:

The task is self-contained (writing, brainstorming, planning)
You can provide the needed context by pasting or uploading files
You are working with sensitive production systems (connect to dev/staging only)
The MCP server adds latency that slows your workflow

MCP Security Considerations

MCP servers run locally and can access real systems. Best practices:

Only connect to development or staging environments, never production
Use read-only database credentials when possible
Limit filesystem access to specific directories using the server's configuration
Review Claude's MCP calls before approving actions that modify data
Use environment variables for credentials rather than hardcoding them in configuration
Audit your MCP server configurations periodically to remove servers you no longer use

Choosing the Right MCP Servers

Not every project needs every MCP server. Start with the minimum set and add more as your workflow demands:

Solo developers: Filesystem + database (if applicable) Frontend developers: Filesystem + browser automation (Playwright) Backend developers: Filesystem + database + API testing Full-stack teams: Filesystem + database + Git + project management

Adding servers you do not actively use wastes Claude's attention. Each connected server expands the list of available tools Claude must evaluate for every request.

Computer Use: Desktop-Level Interaction

Computer Use (currently in beta) allows Claude to interact with your desktop environment by capturing screenshots, controlling the mouse, and providing keyboard input. This enables Claude to use applications that do not have APIs or MCP servers.

When Computer Use Helps with Context

Computer Use is a context-gathering tool in addition to being an interaction tool. Sometimes the easiest way to give Claude context is to let it look at what you are looking at:

GUI applications: Show Claude your IDE, database tools, or monitoring dashboards
Web applications: Let Claude navigate internal tools that require authentication
Design tools: Have Claude reference designs in Figma or Sketch directly
Spreadsheets: Let Claude read complex Excel layouts that do not convert cleanly to CSV

Practical Workflow

Ask Claude to take a screenshot of the current screen
Claude analyzes the visual context and incorporates it into the conversation
You can direct Claude to interact with specific UI elements

This is particularly useful when the relevant context is in a visual format that is difficult to describe in text.

Computer Use Limitations

Computer Use is slower than MCP-based interactions because it relies on visual processing rather than structured data exchange. Use it as a fallback for tools that lack MCP servers or APIs, not as your primary context mechanism. For anything that can be done through MCP (database queries, file access, API calls), MCP is faster and more reliable.

Local File Access: Eliminating the Upload Cycle

Claude Desktop can read from and write to your local file system directly (via MCP filesystem server), eliminating the need to manually upload and download files.

Advantages Over Web Uploads

No file size workarounds: Access files of any size without upload limits
Live files: Claude reads the current version of a file, not a snapshot uploaded hours ago
Write capability: Claude can save outputs directly to your file system
Directory browsing: Claude can explore project structures to understand organization

Best Practices for Local File Access

Scope access narrowly. Point the filesystem MCP server at the specific project directory, not your home folder.
Use it for exploration. Let Claude browse your project structure to build understanding, then focus on specific files.
Combine with Projects. Use Project instructions to set context and local file access to provide the actual content. This gives Claude both the "how" (instructions) and the "what" (files).

External Documents: PDFs and Markdown in Claude Desktop

Claude Desktop handles external documents the same way as Claude Web: through Project knowledge files and conversation uploads. However, the addition of local file access changes the strategy.

The Hybrid Approach

For persistent reference material: Upload to Project knowledge files (PDFs or Markdown). These are always available in every conversation within the Project.

For working documents: Access via the filesystem MCP server. This way Claude reads the live version of your files without requiring re-uploads when content changes.

For published specifications: Upload PDFs to Project knowledge files. These do not change, so the snapshot approach works fine.

For your own documentation: Keep it in Markdown files on disk and access via MCP. This way both you and Claude are always working with the latest version.

Building an Effective Claude Desktop Workflow

Step 1: Set Up Your Project

Create a Claude Desktop Project with:

Project Instructions covering your role, style, constraints, and terminology
Knowledge files for stable reference material (style guides, specifications, standards)

Step 2: Configure MCP Servers

Add MCP servers for the external systems you work with regularly:

Filesystem server pointing at your project directory
Database server connected to your development database (if applicable)
Any service-specific MCP servers for tools you use daily

Step 3: Use the Right Tool for Each Context Need

Context Need	Best Approach
Project conventions and style	Project Instructions
Stable reference documents	Project Knowledge Files
Current code and config files	Filesystem MCP
Database state and schema	Database MCP
Visual UI or application state	Computer Use
One-off data or examples	Paste in conversation

Step 4: Manage Conversation Threads

Even with MCP and local file access, conversation management matters:

Start new conversations for new topics (Project context persists)
Use artifacts for important outputs you want to reference later
Summarize progress when starting fresh threads

Advanced Patterns

The Live Debugging Pattern

When debugging an issue:

Let Claude read the relevant source code via filesystem MCP
Let Claude query the database to check data state
Let Claude read log files to identify error patterns
Have a conversation where Claude synthesizes all of this context into a diagnosis

This approach gives Claude real-time access to a broader context than you could reasonably paste into a conversation.

The Document Generation Pipeline

For creating documents that reference live data:

Claude reads data via MCP (database stats, API responses, configuration)
Claude generates the document in a conversation
Claude writes the output directly to a file on disk
You review and iterate

This eliminates the copy-paste cycle between Claude and your file system.

The Research and Synthesis Pattern

For research projects spanning multiple sources:

Upload academic papers and specifications as Project knowledge files
Connect a web-search MCP server for current information
Use filesystem MCP to read your existing notes and drafts
Claude synthesizes across all sources, referencing each by name

Common Mistakes

Connecting production databases. Always use development or staging credentials. Even read-only production access introduces risk.
Over-scoping filesystem access. Do not give Claude access to your entire home directory. Point the filesystem server at the specific project folder.
Using MCP for everything. If you just need Claude to reference a style guide, upload it to Project knowledge files. MCP is for live, changing data.
Forgetting Project Instructions. MCP and local file access do not replace the need for clear instructions. Claude still needs to know your style, constraints, and output format.
Not reviewing MCP actions. When Claude performs actions through MCP (writing files, running queries), review them. The protocol provides transparency, but you need to exercise your approval authority.

Go Deeper

To learn more about context management strategies for AI tools and agentic workflows, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for Claude Web: A Complete Guide to Projects, Artifacts, and Intelligent Context

Alex Merced — Sat, 07 Mar 2026 11:00:00 GMT

Claude's web interface at claude.ai combines one of the largest context windows in the industry with a structured Project system that makes it genuinely useful for sustained, complex work. While many AI chat interfaces are limited to one-off conversations, Claude Web is designed for ongoing engagement where the AI accumulates understanding of your work over time. The key to unlocking that potential is managing context deliberately rather than treating each conversation as a blank slate.

This guide covers every context management strategy available in Claude Web, from basic conversation techniques to advanced Project workflows that make Claude function as a persistent research and development partner.

How Claude Web Handles Context

Claude Web uses the conversation thread as its primary context unit. Every message you send, every response Claude generates, every file you upload, and every artifact Claude creates stays in the conversation's context window. Models like Claude Sonnet 4.5 and Opus 4.6 support context windows up to 1 million tokens, which means Claude can hold the equivalent of roughly 750,000 words of conversation, documents, and code in memory at once.

But a large context window does not eliminate the need for context management. In fact, it makes it more important. With 1 million tokens available, it is easy to fill the window with irrelevant information that dilutes Claude's attention. The goal is not to maximize how much context you provide, but to maximize how relevant that context is.

The Context Priority Hierarchy

Claude pays the most attention to:

System instructions (Project instructions)
The most recent messages in the conversation
Uploaded files referenced in the conversation
Earlier conversation history

This means that if important context appeared 50 messages ago, Claude may not weight it as heavily as something you said in the last 3 messages. Understanding this hierarchy helps you decide when to re-state important constraints versus trusting that Claude still has them in context.

Thinking About the Right Level of Context

Quick Questions (Minimal Context)

For factual questions, brainstorming, or one-off tasks, just ask. Claude's training data provides sufficient background for most general-knowledge queries. Adding unnecessary context ("I am a senior engineer with 15 years of experience, and I have a question about Python lists") wastes tokens and does not improve the response.

Focused Work (Moderate Context)

For drafting, editing, code review, or analysis, provide the specific material Claude needs to work with. Paste the code you want reviewed, the text you want edited, or the data you want analyzed. State your requirements clearly: what format you want, what constraints apply, what style to follow.

Extended Projects (Comprehensive Context)

For ongoing work spanning multiple conversations, use Claude's Projects feature. Upload reference documents, set Project instructions, and let Claude maintain continuity across sessions. This is where context management becomes a genuine productivity multiplier.

Projects: Claude Web's Most Powerful Context Tool

Projects create persistent workspaces that carry context across conversations. When you create a Project, you define instructions and upload knowledge files that apply to every conversation within that Project.

Setting Up a Project

Navigate to Projects in the Claude sidebar
Create a new Project with a descriptive name
Add Project Instructions: Custom system-level instructions that Claude follows in every conversation within this Project
Upload Knowledge Files: Documents that Claude can reference across all conversations in the Project

Project Instructions

Project instructions function as a system prompt that persists across every conversation in the Project. This is the most important piece of context you configure, because it shapes every response Claude gives.

Effective Project Instructions include:

# Project: Data Pipeline Documentation

## Your Role
You are a technical writer helping document a real-time data pipeline
built with Apache Kafka, Apache Flink, and Apache Iceberg.

## Audience
The documentation is for data engineers with 2-5 years of experience
who are familiar with batch ETL but new to stream processing.

## Style Requirements
- Use active voice
- Include code examples in Python and SQL
- Explain concepts before showing implementation
- Each section should be self-contained (readers may jump between sections)

## Terminology
- Use "data pipeline" not "ETL pipeline" or "data flow"
- Use "event" not "message" when referring to Kafka records
- Use "table" not "dataset" when referencing Iceberg tables

## Output Format
- Use H2 for section headers, H3 for subsections
- Include a "Key Takeaways" box at the end of each section
- Code blocks should include language identifiers

Knowledge Files

You can upload various file types as project knowledge:

File Type	Best For	Notes
PDF	Research papers, specs, published docs	Claude extracts text; complex layouts may lose formatting
Markdown	Style guides, outlines, structured notes	Cleanest parsing, best for AI consumption
Text	Code files, logs, configuration	Direct text ingestion
CSV	Data samples, reference tables	Claude can analyze and query the data
Images	Diagrams, screenshots, mockups	Claude can describe and reference visual content

When to Use PDFs vs. Markdown

Use PDFs when:

You have published documents that already exist in PDF format
The document includes complex tables, figures, or formatting that matters
You do not want to spend time converting the document

Use Markdown when:

You are creating a context document specifically for Claude
You want maximum parsing accuracy (no PDF extraction artifacts)
The document will be updated frequently
You care about precise structure (headings, code blocks, lists)

Markdown is the better choice when you have the option. PDF extraction can introduce artifacts: garbled tables, merged paragraphs, lost code formatting. If accuracy matters, convert your reference documents to Markdown.

Managing Knowledge Files Effectively

Name files descriptively. "api-reference-v3.md" is better than "document.pdf"
Add a summary at the top of each file. Claude can navigate large files more effectively when they start with an overview.
Keep files focused. Five 20-page documents work better than one 100-page document, because Claude can identify which file is relevant to a specific question.
Remove outdated files. Stale information in your knowledge base leads to stale responses.

Artifacts: Context That Claude Creates

Artifacts are a distinct Claude Web feature where Claude creates standalone documents, code files, diagrams, or interactive components during a conversation. Unlike regular responses, artifacts persist as discrete objects that you can reference, edit, and reuse.

How Artifacts Enhance Context Management

Artifacts serve as shared reference points between you and Claude. When Claude creates a code artifact, for example, both of you can reference it by name in subsequent messages. This is more efficient than scrolling through conversation history to find the relevant code block.

Common artifact types:

Code files: Complete, runnable code that Claude creates and iterates on
Documents: Formatted text (reports, drafts, plans) that can be edited in place
Diagrams: Mermaid or SVG diagrams that visualize architectures or workflows
Interactive components: React components that render in the browser

Using Artifacts for Context Persistence

When working on a complex deliverable, ask Claude to create artifacts for each component. This keeps the working documents visible and accessible without being buried in conversation history. You can then reference specific artifacts ("Update the database schema artifact to include the new user_preferences table") rather than re-describing what you need.

MCP Server Support on Claude Web

Claude Web supports MCP (Model Context Protocol) through remote MCP servers. This allows the web interface to connect to external tools and data sources without requiring a local desktop application.

How MCP Works on Claude Web

To connect a remote MCP server:

Navigate to Settings > Connectors in the Claude web interface
Add a custom connector by providing the remote MCP server's URL
The MCP server's tools become available within your conversations

Claude Web supports remote MCP servers across all plan tiers (Free, Pro, Max, Team, Enterprise), though free users may have limitations on the number of connections.

What MCP Enables on Claude Web

With MCP connectors, Claude Web can interact with external services directly:

Productivity tools: Google Drive, Slack, Asana, monday.com
Developer tools: GitHub, Sentry, Linear
Creative tools: Canva, Figma
Custom APIs: Any service exposed through a remote MCP server

MCP Apps

Claude Web also supports MCP Apps, an extension of the protocol that allows MCP servers to provide interactive user interfaces directly within the Claude interface. This means tools connected via MCP can render visual components (dashboards, project boards, design canvases) inside your Claude conversation, reducing the need to switch between applications.

Claude Web vs. Claude Desktop MCP

Claude Web connects to remote MCP servers (cloud-hosted, accessed via URL). Claude Desktop supports both remote and local MCP servers (processes running on your machine via STDIO). If you need to connect to local databases, local file systems, or services that are not exposed to the internet, use Claude Desktop. For cloud-hosted services and APIs, Claude Web's remote MCP support is sufficient.

Structuring Context for Maximum Impact

The Briefing Pattern

At the start of a new conversation within a Project, briefly re-state the current focus:

"We are working on Chapter 3 of the documentation, covering Flink job deployment. The outline is in the project files. I want to draft the section on checkpoint configuration."

This grounds Claude immediately without requiring it to search through the full conversation history or project files.

The Explicit Reference Pattern

When you want Claude to use specific information from your project files, reference them directly:

"Based on the API reference document I uploaded, write example code that demonstrates the batch ingestion endpoint. Follow the code style shown in the style guide document."

Explicit references help Claude prioritize the right source material rather than relying on its general knowledge.

The Iterative Refinement Pattern

For complex outputs, work in stages:

Outline first: "Create an outline for this section covering X, Y, and Z"
Draft section by section: "Write the first section based on the outline"
Review and refine: "The technical content is good but the tone is too formal. Make it conversational."
Consistency check: "Review the full draft for consistency in terminology and style"

Each stage keeps Claude's focus narrow, which produces better results than asking for a complete deliverable in one shot.

Managing Long Conversations

Even with a 1-million-token context window, very long conversations can degrade quality. When a conversation starts feeling unfocused:

Start a new conversation within the same Project (your files and instructions carry over)
Summarize progress at the start of the new conversation
Create artifacts for important outputs so they are easy to reference in the new thread

Advanced Patterns

The Multi-Perspective Analysis

Ask Claude to analyze a problem from multiple angles in a single conversation:

"First, analyze this architecture from a performance perspective. Then, analyze it from a cost perspective. Finally, analyze it from a maintainability perspective. Structure each analysis as a separate section."

This leverages Claude's large context window to produce comprehensive analysis while keeping the output organized.

The Living Document Workflow

Use a Project with a master document artifact that Claude updates throughout the engagement:

Create an initial artifact (e.g., "Project Plan v1")
As work progresses, ask Claude to update the artifact
The artifact becomes a living record of the project's evolution

This is particularly effective for research, planning, and documentation work.

The Expert Panel Pattern

Give Claude multiple "hats" to wear within a Project:

"In this Project, I want you to evaluate ideas from three perspectives: (1) a cautious security engineer, (2) an enthusiastic product manager, and (3) a pragmatic senior developer. When I present an idea, respond with all three perspectives."

This turns a single Claude conversation into a simulated review process.

Common Mistakes

Not using Projects for project work. If you have more than 3 conversations about the same topic, you should be using a Project. Without it, you lose continuity between sessions.
Uploading too many files without organization. Quality beats quantity. Upload the files Claude actually needs, name them well, and include summaries.
Ignoring Project Instructions. Many users create Projects but skip the instructions. This is like hiring a consultant but never briefing them. The instructions are the single highest-impact piece of context you can provide.
Not starting fresh conversations. Long conversations accumulate noise. When you shift to a new subtopic, start a new conversation within the Project.
Not connecting MCP servers when you need live data. Claude Web supports remote MCP servers through Settings > Connectors. If your task requires live connections to cloud services, set up the relevant MCP connectors. For local services not exposed to the internet, use Claude Desktop instead.

Go Deeper

To learn more about context management strategies for AI tools and agentic workflows, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for OpenAI Codex: A Complete Guide Across Browser, CLI, and App

Alex Merced — Sat, 07 Mar 2026 10:00:00 GMT

OpenAI Codex is not a chatbot. It is an autonomous software engineering agent that runs tasks in isolated cloud sandboxes, operates across a browser interface, a command-line tool, and a dedicated macOS app, and can work on multiple tasks in parallel. Because of this architecture, context management in Codex works fundamentally differently from ChatGPT or traditional coding assistants. Instead of conversational context windows, you manage context through persistent configuration files, skill definitions, and project-level instructions that shape how the agent approaches your codebase.

This guide covers every context management mechanism Codex provides, explains when to use each one, and walks through practical strategies for getting the agent to produce reliable, project-aligned results across all three interfaces.

Understanding How Codex Handles Context

Codex operates with a large context window (approximately 192,000 tokens), which means it can reason about substantial portions of a codebase in a single task. But context in Codex is not just conversation history. The agent assembles its context dynamically from multiple sources:

Your repository: Codex clones your repo into a sandboxed environment for each task
AGENTS.md files: Persistent instructions that live in your repository
Skills: Reusable bundles of instructions, templates, and scripts
Task prompt: Your natural language description of what to do
Previous interactions: In the desktop app, persistent project memory carries context across sessions

The key insight is that most of Codex's context comes from your repository itself, not from conversational back-and-forth. This makes context management a matter of preparing your repo and configuration files rather than crafting perfect prompts.

Thinking About the Right Level of Context

Minimal Context (Quick Tasks)

For simple, self-contained tasks like "add input validation to this function" or "write unit tests for utils.py," the task prompt and the codebase itself provide sufficient context. Codex will explore the relevant files, understand the patterns, and produce targeted changes. You do not need to provide extensive background.

Moderate Context (Targeted Changes)

For tasks that require understanding project conventions, architectural decisions, or specific technical requirements, provide that context in your AGENTS.md file or in the task prompt. For example: "Refactor the authentication module to use JWT instead of session cookies. Our API follows REST conventions and uses Express 5 middleware patterns."

Comprehensive Context (Large Features or Ongoing Work)

For multi-step features, large refactors, or ongoing development work, invest in Skills and detailed AGENTS.md files. These provide the agent with your coding standards, architectural patterns, testing requirements, and deployment constraints. The desktop app's persistent project memory also helps here by retaining context across sessions.

AGENTS.md: The Foundation of Codex Context

AGENTS.md is the most important context management tool for Codex. It is a Markdown file that lives in your repository and provides persistent instructions to the agent. Codex reads AGENTS.md at the beginning of every task.

How It Works

Place an AGENTS.md file at the root of your repository. Codex loads it automatically before starting any task. Think of it as a briefing document that tells the agent everything it needs to know about your project.

What to Include

# AGENTS.md

## Project Overview
This is a Next.js 15 application with a Python FastAPI backend.
The frontend uses TypeScript, Tailwind CSS, and Zustand for state management.
The backend uses SQLAlchemy with PostgreSQL.

## Coding Standards
- Use functional components with hooks (no class components)
- All API endpoints must include input validation using Pydantic
- Write tests for every new function using pytest (backend) and Vitest (frontend)
- Use conventional commit messages: feat:, fix:, refactor:, docs:, test:

## Architecture
- Frontend routes are in src/app/ (App Router)
- API routes are in backend/api/routes/
- Database models are in backend/models/
- Shared types are in shared/types/

## Constraints
- Do not modify the database schema without explicit approval
- Do not add new dependencies without noting them in the PR description
- All environment variables must be documented in .env.example

Hierarchical AGENTS.md Files

For monorepos or large projects, you can place AGENTS.md files at different levels:

Root level: Global project instructions
Service directories: Service-specific conventions (e.g., backend/AGENTS.md, frontend/AGENTS.md)
Global: ~/.codex/AGENTS.md for personal preferences that apply across all projects

More specific files supplement (not replace) more general ones. The agent combines all applicable AGENTS.md files when executing a task.

Best Practices

Keep it updated. Stale AGENTS.md instructions lead to stale agent behavior.
Be specific about constraints. "Follow best practices" is meaningless to an agent. "All database queries must use parameterized statements, never string interpolation" is actionable.
Include examples of your code style. Show the agent what "good" looks like in your codebase.
Document your testing strategy. Tell the agent which test framework to use, where tests live, and what coverage expectations you have.

Skills: Reusable Workflow Bundles

Skills are a step beyond AGENTS.md. They are reusable bundles that package instructions, code templates, API configurations, and scripts into a single invocable unit. Skills let you codify complex workflows so the agent can execute them reliably.

When to Use Skills

You have a repeatable workflow (deploying to staging, onboarding a new API endpoint, migrating a database)
The workflow requires multiple steps that need to happen in a specific order
You want consistency across team members using Codex

Creating a Skill

Skills are defined as structured folders with a manifest file:

# SKILL.md

---
name: create-api-endpoint
description: Creates a new REST API endpoint with validation, tests, and documentation
---

## Steps
1. Create the route file in backend/api/routes/
2. Define the Pydantic request/response models in backend/api/schemas/
3. Implement the business logic in backend/services/
4. Write pytest tests in backend/tests/
5. Add the endpoint to the OpenAPI documentation
6. Update the API changelog

## Templates
Use the existing endpoint at backend/api/routes/users.py as the reference pattern.

## Validation
- Run pytest after creating the endpoint
- Verify the OpenAPI spec is valid
- Check that all response codes are documented

Skills can be invoked explicitly by name or triggered automatically when the agent detects a task that matches the skill's description.

The Three Interfaces: Context Differences

Browser (ChatGPT Sidebar)

The browser interface runs Codex from within the ChatGPT web application. Context management here is straightforward:

Repository: Select which repo the agent works on
Task prompt: Describe what you want done
AGENTS.md: Loaded automatically from the repo
Results: The agent produces a diff or pull request for review

This interface is best for individual tasks that you want to review before merging. Context is session-scoped; each task gets a fresh sandbox.

CLI (Command Line)

The Codex CLI (codex) runs in your terminal and operates on your local codebase. It offers more control over context:

Approval modes: Choose between Chat (interactive), Agent (approval for writes), and Full Access (autonomous)
MCP servers: The CLI supports MCP server integration for connecting external tools
File references: Point the agent at specific files or directories
Image inputs: Pass screenshots or design mockups alongside prompts
Interactive mode: Have a conversation with the agent about your codebase

The CLI is the most flexible interface for context management because you can combine AGENTS.md, MCP servers, and direct file references in a single session.

Desktop App (macOS)

The desktop app is the most powerful interface for sustained work:

Persistent project memory: The app retains project history and context across sessions, so you do not have to re-establish context every time
Multi-agent orchestration: Run multiple agents on different tasks simultaneously, each in its own Git worktree
Visual task management: See all running and completed tasks in a unified interface
Skills management: Create, organize, and invoke Skills from the app

The desktop app is best for ongoing development work where you are regularly delegating tasks to Codex throughout your day.

MCP Server Support

The Codex CLI supports the Model Context Protocol (MCP), allowing you to connect external tools and data sources to the agent.

What MCP Enables

Database access: Let the agent query your development database to understand schema and data patterns
Browser automation: Connect a Playwright MCP server so the agent can test frontend changes by interacting with a real browser
API integration: Give the agent access to your project management tools, documentation systems, or monitoring dashboards
Custom tools: Build MCP servers that expose your organization's internal tools to the agent

When to Use MCP

MCP is most valuable when the agent needs information that is not in the repository:

Understanding runtime behavior (logs, database state, API responses)
Verifying changes against a running application
Accessing external specifications or documentation
Interacting with CI/CD systems or deployment tools

When NOT to Use MCP

For tasks that are purely code-level (refactoring, writing tests, fixing type errors), MCP adds unnecessary complexity. The codebase itself provides sufficient context. Use MCP when the agent needs to interact with the world outside the code.

Configuration

MCP servers are configured through the CLI:

# Add a Playwright MCP server for browser testing
codex mcp add playwright

# Add a custom database MCP server
codex mcp add my-db-server --command "node /path/to/db-mcp.js"

External Documents: When to Use PDFs vs. Markdown

Codex primarily operates on code, but there are situations where providing external documents improves results.

Use Markdown When:

Writing AGENTS.md or Skills (required format)
Providing architectural decision records (ADRs)
Sharing coding standards or style guides
Documenting API specifications

Markdown is the native format for Codex context. It parses cleanly, supports code blocks, and is version-controllable in Git.

Use PDFs When:

Referencing published specifications (RFC documents, protocol specs)
Sharing design documents with diagrams that do not translate well to Markdown
Providing compliance or regulatory requirements that exist in PDF form

In practice, Markdown is almost always the better choice for Codex. If you have a PDF specification, consider extracting the relevant sections into a Markdown file in your repository.

Automations: Scheduled Context Processing

Codex supports Automations, which are scheduled tasks that run in the background. These allow you to set up recurring agent work that automatically processes your codebase with predefined context.

Use Cases

Daily code reviews: Schedule the agent to review new PRs every morning
Dependency audits: Weekly check for outdated or vulnerable dependencies
Documentation updates: Automatically update API documentation after code changes
Test maintenance: Periodically scan for broken or flaky tests

Automations use the same AGENTS.md and Skills context as manual tasks, ensuring consistency between scheduled and ad-hoc work.

Advanced Patterns

The Context Layering Strategy

Combine multiple context sources for complex tasks:

Global AGENTS.md (in ~/.codex/): Personal preferences and universal standards
Project AGENTS.md (in repo root): Project architecture and conventions
Directory AGENTS.md (in subdirectories): Component-specific patterns
Skills: Repeatable workflows for common tasks
Task prompt: The specific thing you want done now
MCP servers: Live external data for verification

Each layer adds specificity without overriding the layers above it.

The Multi-Agent Pattern

Use the desktop app to run parallel agents on different aspects of a feature:

Agent 1: Implements the backend API endpoint
Agent 2: Writes the frontend component
Agent 3: Creates integration tests

Each agent runs in its own Git worktree, so their changes do not conflict. Review and merge the results when all agents complete.

The Exploration-First Pattern

Before giving Codex a complex task, use a "planning" prompt:

"Analyze the authentication module in backend/auth/. Describe the current architecture, identify potential issues, and suggest improvements. Do not make any changes."

Review the agent's analysis, then use it as context for the actual implementation task. This prevents the agent from making changes based on incomplete understanding.

Common Mistakes

Skipping AGENTS.md: Without AGENTS.md, the agent has no guidance on project conventions and will produce code that technically works but does not match your style.
Overly broad tasks: "Improve the application" is too vague. "Add rate limiting to the /api/users endpoint using express-rate-limit with a 100-request-per-minute window" gives the agent clear parameters.
Ignoring the review step: Codex produces diffs and PRs for a reason. Always review the output, especially for tasks involving security, database changes, or public-facing features.
Not using Skills for repeatable work: If you find yourself writing the same type of task prompt repeatedly, extract it into a Skill.
Using MCP when you do not need it: Adding MCP servers increases complexity and potential failure points. Only connect external tools when the task genuinely requires external data.

Go Deeper

To learn more about working effectively with AI coding tools, context engineering, and agentic development workflows, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

Context Management Strategies for ChatGPT: A Complete Guide to Getting Better Results

Alex Merced — Sat, 07 Mar 2026 09:00:00 GMT

Getting consistently useful results from ChatGPT requires more than writing good prompts. The real differentiator is how you manage context: the background information, instructions, documents, and accumulated knowledge that shapes every response ChatGPT generates. Without deliberate context management, you end up repeating yourself, getting generic answers, and wasting time course-correcting the AI.

This guide covers every context management tool ChatGPT offers in 2026, from basic custom instructions to advanced Project workflows, and explains when to use each one.

What Is Context Management and Why Does It Matter?

Context management is the practice of controlling what information an AI model has access to when generating a response. Every time you interact with ChatGPT, the model processes a "context window," basically the sum of all text it can see at once, including your conversation history, uploaded files, system instructions, and memory. The quality of the response depends directly on how well you curate that window.

Poor context management looks like this:

Repeating your role, preferences, and constraints in every new conversation
Uploading the same reference documents over and over
Getting responses that ignore your project's specific terminology or conventions
Spending more time correcting the AI than doing actual work

Good context management means ChatGPT already knows your background, has access to relevant documents, follows your preferred style, and builds on previous conversations without you manually re-establishing all of that every time.

Thinking About the Right Level of Context

Before configuring any tools, think about what level of context a given task actually needs. Not every conversation requires the same depth.

Minimal Context (Quick Questions)

For simple factual questions, brainstorming, or one-off tasks, you often need zero setup. Just ask the question. Adding unnecessary context actually dilutes the model's attention and can lead to worse responses. If you are asking "What is the difference between TCP and UDP?" you do not need to upload your network architecture docs.

Moderate Context (Focused Work)

For tasks like drafting emails, reviewing code snippets, or writing sections of a document, provide the immediately relevant information in the conversation. Paste the specific text you are working with, reference the specific style or tone you want, and state any constraints. This keeps the model focused without overwhelming it.

Comprehensive Context (Extended Projects)

For ongoing projects, research, or multi-session work, use ChatGPT's structured context tools (Projects, CustomGPTs, Memory). This is where deliberate context management pays the biggest dividends. You define the context once and it persists across every conversation in that workspace.

How to Decide

Ask yourself: "If I handed this task to a knowledgeable colleague, what would I need to tell them before they could start?" If the answer is "nothing, just the question," use minimal context. If you would need to hand them a style guide, a codebase overview, and three reference documents, set up a Project.

Custom Instructions: Your Global Defaults

Custom Instructions are the most basic and most overlooked context management tool in ChatGPT. They apply to every conversation you have (unless you use a Project or CustomGPT with its own instructions).

How They Work

Navigate to Settings > Personalization > Custom Instructions. You get two fields:

About you: Tell ChatGPT who you are, what you do, and what background knowledge to assume. For example: "I am a senior data engineer working with Apache Iceberg, Spark, and Python. I build data lakehouse architectures for financial services companies."
How to respond: Define your preferred output format, tone, and constraints. For example: "Be concise. Use code examples in Python unless I specify otherwise. Skip basic explanations of concepts I already know. Never use em dashes."

Best Practices

Keep instructions specific and actionable. "Be helpful" is useless. "When I ask about SQL, always format queries with uppercase keywords and include comments explaining each join" is useful.
Update them as your needs change. If you switch projects or roles, update your instructions.
Use negative constraints. Telling ChatGPT what NOT to do is often more effective than listing everything it should do.
Do not overload them. Custom Instructions have a character limit. Use them for universal preferences, not project-specific details.

Limitations

Custom Instructions are global. They apply everywhere unless overridden by a Project or CustomGPT. If you work across multiple domains (coding, writing, research), your instructions need to be general enough to help everywhere without being so vague they help nowhere. For domain-specific work, use Projects instead.

Memory: Persistent Knowledge Across Conversations

ChatGPT's Memory feature allows the model to remember facts, preferences, and context across conversations without you re-stating them.

How Memory Works

When enabled (Settings > Personalization > Memory), ChatGPT can save information you share during conversations. It stores these as discrete facts: "User prefers Python over JavaScript," "User's company uses PostgreSQL 15," "User is writing a book about data engineering."

You can explicitly tell ChatGPT to remember things: "Remember that my team uses the Google style guide for Python." You can also ask it what it remembers ("What do you know about me?") and delete specific memories or clear them all.

When to Use Memory

Memory is best for facts that apply broadly across conversations:

Your technical stack and preferences
Your role and expertise level
Recurring project names or team members
Style preferences that should persist everywhere

When NOT to Use Memory

Memory is not a substitute for Projects or file uploads. It stores brief facts, not documents or complex context. Do not try to make ChatGPT "memorize" an entire API specification through Memory. Use file uploads for that.

Temporary Chats

If you want a conversation without Memory recall (for example, helping someone else with their problem or exploring a sensitive topic), use Temporary Chat. This creates a blank-slate conversation that does not read from or write to Memory.

Projects: Dedicated Workspaces for Focused Work

Projects are ChatGPT's most powerful context management feature for sustained work. A Project is a dedicated workspace that groups related conversations, uploaded files, and custom instructions.

Setting Up a Project

Click Projects in the sidebar
Create a new Project with a descriptive name
Add Project Instructions: These override or supplement your global Custom Instructions for every conversation within this Project
Upload files: Up to 20 files per Project (PDFs, CSVs, images, text files). ChatGPT can reference these across all conversations in the Project.
Start conversations within the Project

Project Instructions vs. Custom Instructions

Project Instructions are scoped to the Project. They are the right place for:

Project-specific terminology and conventions
The structure or outline of what you are building
Style guides or formatting requirements specific to this work
Background context about the domain

Think of Custom Instructions as your personal defaults and Project Instructions as the briefing document for a specific engagement.

File Management in Projects

You can upload various file types to a Project's knowledge base:

File Type	Best For
PDF	Reference documentation, research papers, specifications
CSV/Excel	Data samples, structured reference data
Text/Markdown	Style guides, code snippets, outlines, notes
Images	Diagrams, mockups, screenshots for visual context

When to Use PDFs vs. Markdown

This is a practical question that matters more than most people realize.

Use PDFs when:

The document is a published specification, whitepaper, or research paper
Layout and formatting matter (tables, figures, page references)
You have the document in PDF form and do not want to convert it

Use Markdown when:

You are creating context documents specifically for ChatGPT
You want the AI to parse the content with maximum accuracy
The content is structured text (code standards, API docs, outlines)
You plan to update the document frequently

Markdown is generally a better format for AI consumption. The structure is unambiguous, there are no encoding issues from PDF extraction, and the content is more reliably parsed. If you are creating a reference document from scratch to guide ChatGPT, write it in Markdown.

Project Sharing

Projects can be shared with other ChatGPT users. When you share a Project, collaborators get access to the uploaded files, Project Instructions, and conversation history. This makes Projects useful for team workflows where multiple people need the AI to have the same context.

CustomGPTs: Specialized Assistants for Repeatable Tasks

CustomGPTs let you create purpose-built AI assistants with specific instructions, knowledge bases, and capabilities. They are the right tool when you have a repeatable workflow that requires specialized context.

When to Use a CustomGPT vs. a Project

Feature	Project	CustomGPT
Best for	Extended work on a specific project	Repeatable tasks across different projects
Context scope	One body of work	One type of task
Shareable	Yes (collaborators)	Yes (public or private)
Custom actions	No	Yes (API integrations)
Example	"Q3 Marketing Campaign"	"Technical Blog Editor"

A CustomGPT is like hiring a specialist. A Project is like setting up a war room for a specific mission.

Building an Effective CustomGPT

Instructions: Write detailed behavioral instructions. Include the role, tone, output format, and constraints. Be as specific as your best Custom Instructions, but scoped to this GPT's purpose.
Knowledge files: Upload reference documents that the GPT should always have access to. These function like Project files but are permanently attached to the GPT.
Actions: Connect external APIs so the GPT can fetch real-time data, submit forms, or interact with your tools.

Knowledge Base Best Practices

Name your files descriptively. "company-style-guide-2026.md" is better than "doc1.pdf."
Include a table of contents or summary at the top of large documents. This helps ChatGPT navigate the content.
Keep individual files focused. Ten small, focused files work better than one 200-page PDF.
Test your GPT after uploading. Ask questions that require it to reference specific sections of your documents to verify it is parsing them correctly.

MCP Server Support

ChatGPT added support for the Model Context Protocol (MCP) in September 2025 through a "Developer Mode" feature. This is available to paying users on Plus, Pro, Team, Enterprise, and Education plans.

How MCP Works in ChatGPT

With Developer Mode enabled, ChatGPT can connect to MCP servers that expose external tools and data sources. This means ChatGPT can interact with services like Jira, Google Calendar, databases, and custom APIs directly from the chat interface. MCP connections are configured through the ChatGPT settings under Developer Mode, where you specify the MCP server endpoints.

What MCP Enables

MCP in ChatGPT goes beyond read-only data access. It supports both read and write operations, meaning ChatGPT can:

Fetch data from external systems (database queries, API lookups)
Update external systems (create tickets, send messages, update records)
Interact with local files and applications when using the desktop app

MCP vs. CustomGPT Actions

Feature	MCP Servers	CustomGPT Actions
Protocol	Standardized (MCP)	Custom API definitions
Setup	Configure via Developer Mode	Build into a CustomGPT
Portability	Works across MCP-compatible tools	ChatGPT only
Operations	Read and write	Read and write

MCP servers offer the advantage of portability: the same MCP server you use with ChatGPT works with Claude Desktop, Cursor, and other MCP-compatible tools. CustomGPT Actions are ChatGPT-specific but offer tighter integration within the CustomGPT workflow.

Security Considerations

OpenAI has cautioned that using Developer Mode with write operations is powerful but carries risk. Always test MCP server connections carefully, especially for servers that can modify external systems. Be aware of potential prompt injection risks when connecting to untrusted data sources.

Structuring Context for Maximum Effectiveness

Beyond the tools themselves, how you structure the information you give ChatGPT matters significantly.

The Inverted Pyramid

Put the most important context first. ChatGPT pays more attention to the beginning and end of its context window. Structure your information like a news article:

Lead: The task, constraint, and desired output format
Body: Supporting details, reference material, examples
Background: Nice-to-have context that might help but is not critical

Be Explicit About What You Want

Vague requests get vague results. Compare:

Vague: "Help me with my database." Specific: "Review this PostgreSQL query for performance issues. The table has 50 million rows, is partitioned by date, and has indexes on customer_id and order_date. Suggest index changes or query rewrites that would reduce execution time."

The specific version gives ChatGPT enough context to provide actionable advice. The vague version will produce a generic tutorial.

Use Reference Examples

When you want a specific output format or style, give ChatGPT an example. "Write a commit message in this style: [example]" is far more effective than describing the style in abstract terms. Examples are compressed context. One good example communicates more than a paragraph of description.

Manage Conversation Length

Long conversations degrade response quality. As the conversation history grows, ChatGPT has less room in its context window for your actual question and the reasoning needed to answer it. For extended work:

Start new conversations for new topics, even within the same Project
Summarize progress before starting a new conversation ("Here is where we left off: [summary]")
Use Projects so you do not lose the files and instructions when you start fresh

Advanced Patterns

The Briefing Document Pattern

Create a Markdown file that serves as a comprehensive briefing for ChatGPT. Include:

# Project: [Name]

## Overview
[2-3 sentence summary of what this project is]

## Goals
- [Specific goal 1]
- [Specific goal 2]

## Constraints
- [Technical constraints]
- [Style/format constraints]

## Key Terminology
- **Term 1:** Definition specific to this project
- **Term 2:** Definition specific to this project

## Current Status
[Where the project stands right now]

## What I Need Help With
[Specific areas where ChatGPT should focus]

Upload this as a Project file or paste it at the start of key conversations. It gives ChatGPT a structured, scannable overview that dramatically improves response relevance.

The Iterative Refinement Loop

For complex outputs (long documents, code architectures, research reports):

Start with a high-level outline and get ChatGPT's feedback
Refine the outline based on the feedback
Generate content section by section, reviewing each before moving on
Use follow-up prompts to refine specific sections
Do a final consistency pass

This approach keeps the context focused at each step rather than asking ChatGPT to hold the entire deliverable in mind at once.

Multi-GPT Workflows

For complex projects, use different CustomGPTs for different aspects of the work:

A "Research GPT" with academic papers and data sources
A "Writing GPT" with your style guide and brand voice instructions
A "Code Review GPT" with your codebase standards and architecture docs

Feed the output of one into the next. This keeps each GPT focused on what it does best instead of trying to make one GPT do everything.

Common Mistakes to Avoid

Overloading context: More is not always better. If you upload 20 files but your question only relates to one, the AI may pull irrelevant information from the other 19. Be selective.
Ignoring Custom Instructions: Many users never set them up, then wonder why ChatGPT gives generic responses. Spending 10 minutes on Custom Instructions saves hours of correction.
Not using Projects for project work: Having 50 disconnected conversations about the same project means ChatGPT has no persistent context. Use Projects.
Treating Memory as a database: Memory stores brief facts, not documents. If you need ChatGPT to reference a 30-page specification, upload it as a file.
Never clearing context: Sometimes the best thing to do is start a fresh conversation. If ChatGPT seems confused or is repeating mistakes, the conversation history may be working against you.

Recommended Workflow for New Users

First 10 minutes: Set up Custom Instructions with your role, expertise level, and response preferences
First project: Create a Project, upload 2-3 key reference documents, and write Project Instructions
First week: Enable Memory and let it accumulate useful facts. Review and edit memories periodically.
First month: If you find yourself doing the same type of task repeatedly, build a CustomGPT for it.

Go Deeper

To learn more about working effectively with AI tools, including detailed context management strategies for coding, research, and professional workflows, check out these resources by Alex Merced:

The 2026 Guide to AI-Assisted Development covers AI-assisted development workflows, prompt engineering, and context strategies for software engineers.
The 2026 Guide to Lakehouses, Apache Iceberg and Agentic AI explores how AI agents are reshaping data architecture and how to build systems that support agentic workflows.

And for a fictional take on where AI is heading:

The Emperors of A.I. Valley: A Novel of Power, Code, and the War for the Future is a novel about the power struggles and ethical dilemmas behind the companies building the most powerful AI systems in the world.

How to Use Dremio with Zed: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 21:00:00 GMT

Zed is an open-source, GPU-accelerated code editor written in Rust. It is designed for speed and collaboration, with a built-in AI assistant that supports multiple LLM providers and an agent mode for autonomous multi-step development. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.

Connecting them gives Zed's AI agent the context it needs to write accurate Dremio SQL, generate data pipelines, and build applications against your lakehouse. Zed's performance advantage is significant for data work: its GPU-accelerated rendering handles large result sets and complex code without the lag common in Electron-based editors.

Zed supports MCP through its settings, uses AGENTS.md as its primary context file, and provides agent profiles for scoping tool access to specific workflows.

This post covers four approaches, ordered from quickest setup to most customizable.

Setting Up Zed

If you do not already have Zed installed:

Download Zed from zed.dev (available for macOS and Linux).
Install it by running the installer or using Homebrew: brew install zed.
Configure your AI model in Settings > AI. Zed supports its own hosted models, Anthropic, OpenAI, Google, and Ollama for local models.
Open a project by launching Zed and opening your project directory.

Zed is free and open-source under the GPL license. Its native Rust architecture makes it significantly faster than Electron-based editors, with sub-millisecond input latency and GPU-accelerated rendering.

Approach 1: Connect the Dremio Cloud MCP Server

Every Dremio Cloud project ships with a built-in MCP server. Zed supports MCP through its JSON settings file, where MCP servers are configured as context servers.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup. For Zed, you configure the MCP connection through settings.json.

Find Your Project's MCP Endpoint

Log into Dremio Cloud and navigate to Project Settings > Info. Copy the MCP server URL.

Set Up OAuth in Dremio Cloud

Go to Settings > Organization Settings > OAuth Applications.
Click Add Application and enter a name (e.g., "Zed MCP").
Add the appropriate redirect URIs.
Save and copy the Client ID.

Configure Zed's MCP Connection

Open Zed's settings (Cmd+,) and add the MCP server configuration:

{
  "context_servers": {
    "dremio": {
      "command": {
        "path": "npx",
        "args": ["-y", "@dremio/mcp-client", "--url", "https://YOUR_PROJECT_MCP_URL"]
      }
    }
  }
}

For project-level configuration, create a .zed/settings.json file in your project root with the same structure.

Zed's AI agent now has access to Dremio's MCP tools:

GetUsefulSystemTableNames returns available tables.
GetSchemaOfTable returns column definitions.
GetDescriptionOfTableOrSchema pulls catalog descriptions.
GetTableOrViewLineage shows data lineage.
RunSqlQuery executes SQL and returns results.

Test by opening the agent panel and asking: "What tables are available in Dremio?"

Self-Hosted Alternative

For Dremio Software deployments, use the dremio-mcp server:

{
  "context_servers": {
    "dremio": {
      "command": {
        "path": "uv",
        "args": [
          "run", "--directory", "/path/to/dremio-mcp",
          "dremio-mcp-server", "run"
        ]
      }
    }
  }
}

Approach 2: Use AGENTS.md for Dremio Context

Zed uses AGENTS.md as its primary context file. Place it in your project root and reference it in agent conversations with @agents.md.

Writing a Dremio Context File

Create AGENTS.md in your project root:

# Dremio Project Context

## SQL Conventions
- Use CREATE FOLDER IF NOT EXISTS (not CREATE NAMESPACE or CREATE SCHEMA)
- Tables in the Open Catalog use folder.subfolder.table_name
- External federated sources use source_name.schema.table_name
- Cast DATE to TIMESTAMP for consistent joins
- Use TIMESTAMPDIFF for duration calculations

## Credentials
- Never hardcode Personal Access Tokens. Use environment variable: DREMIO_PAT
- Cloud endpoint: environment variable DREMIO_URI

## Terminology
- Call it "Agentic Lakehouse", not "data warehouse"
- "Reflections" are pre-computed optimizations, not "materialized views"

## Reference
- SQL syntax: ./docs/dremio-sql-reference.md
- Python SDK: ./docs/dremioframe-patterns.md
- Table schemas: ./docs/table-schemas.md

When starting a new agent session, type @agents.md to load the context. Zed will include the file contents in the agent's working context.

Agent Profiles

Zed supports agent profiles for controlling which tools are available. Create a "Dremio Data" profile that enables MCP tools and file editing while restricting terminal access:

In Settings > AI > Profiles, create a profile with specific tool permissions. This is useful for separating data exploration (read-only MCP queries) from development work (full tool access).

Approach 3: Install Pre-Built Dremio Skills and Docs

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill (Community)

The dremio-agent-skill repository provides knowledge files:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cd dremio-agent-skill
./install.sh

Copy the knowledge directory into your project. Reference it in your AGENTS.md:

For Dremio conventions, read the knowledge files in ./dremio-skill/knowledge/.

dremio-agent-md (Community)

The dremio-agent-md repository provides documentation sitemaps:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Reference it in AGENTS.md:

For Dremio SQL validation, read DREMIO_AGENT.md in ./dremio-agent-md/.

Approach 4: Build Your Own AGENTS.md Context

Create a comprehensive context file tailored to your team:

# Team Dremio Data Context

## Environment
- Lakehouse: Dremio Cloud
- Catalog: Apache Polaris-based Open Catalog
- Architecture: Medallion (bronze → silver → gold)

## Table Schemas (updated weekly)
For exact column definitions, read ./docs/table-schemas.md

## SQL Standards
- Bronze: raw.*, Silver: cleaned.*, Gold: analytics.*
- Always use TIMESTAMP, never DATE
- Validate functions against ./docs/dremio-sql-reference.md

## Common Queries
For frequently used patterns, read ./docs/common-queries.md

## Python SDK
- Use dremioframe for all Dremio connections
- Patterns: read ./docs/dremioframe-patterns.md

Zed's fast file loading means referencing external docs adds negligible latency. Keep the AGENTS.md concise and point to detailed reference files.

Using Dremio with Zed: Practical Use Cases

Once Dremio is connected, Zed's AI agent can execute complete data projects with the speed advantage of a native editor.

Ask Natural Language Questions About Your Data

Open the agent panel and ask:

"What were our top 10 products by revenue last quarter? Show growth rates and regional breakdown."

Zed's agent uses MCP to discover tables, writes SQL, and returns results. The GPU-accelerated rendering handles large result tables without lag.

Follow up:

"For products with negative growth, show the correlation between customer complaints and revenue decline over the last 6 months."

The agent maintains context and generates multi-table analytical queries.

Build a Locally Running Dashboard

Ask the agent to create a dashboard:

"Query Dremio gold-layer views for revenue metrics and build an HTML dashboard with Plotly.js. Include monthly trends, regional heatmap, and top customer charts. Add a dark theme, date filters, and export buttons."

The agent generates the complete dashboard across multiple files. Zed's multi-buffer editing lets you see all generated files side-by-side without performance degradation.

Create a Data Exploration App

Build interactive tools:

"Create a Streamlit app connected to Dremio via dremioframe. Include schema browsing, data preview, SQL editor, and CSV download. Generate all files."

The agent generates the full application. Zed's speed makes iterating on the generated code feel instantaneous.

Generate Data Pipeline Scripts

Automate data engineering:

"Write a Medallion pipeline using dremioframe. Bronze ingestion, silver cleaning with deduplication and validation, gold aggregations with business metrics. Include logging and dry-run mode."

The agent writes the pipeline following your AGENTS.md conventions.

Build API Endpoints Over Dremio Data

Create backend services:

"Build a FastAPI app serving Dremio gold-layer data. Add endpoints for analytics, customer segments, and product performance. Include Pydantic models and OpenAPI docs."

The agent generates the complete API server.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time access
AGENTS.md	10 minutes	Convention enforcement, reference file pointers	Teams that want speed + context control
Pre-Built Skills	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Quick start with broad coverage
Custom Context	30+ minutes	Tailored schemas, profiles, and team conventions	Mature teams with specific workflows

Start with the MCP server for live data access. Add AGENTS.md with conventions and reference file pointers. Use agent profiles to scope tool access for different workflows.

Get Started

Sign up for a free Dremio Cloud trial (30 days, $400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Add it to Zed's settings.json under context_servers.
Create AGENTS.md with your Dremio conventions.
Open the agent panel and ask it to explore your Dremio catalog.

Dremio's Agentic Lakehouse gives Zed's agent accurate data context, and Zed's native performance makes data exploration and code generation feel effortless.

For more on the Dremio MCP Server, check out the official documentation or enroll in the free Dremio MCP Server course on Dremio University.

How to Use Dremio with Windsurf: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 20:00:00 GMT

Windsurf is an AI-native code editor built as a fork of VS Code. Its standout feature is Cascade, an agentic AI system that plans and executes multi-step coding tasks autonomously. Cascade understands your entire codebase, can chain together multiple file edits, terminal commands, and tool calls in a single flow. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.

Connecting them gives Cascade the context it needs to write accurate Dremio SQL, generate data pipelines, and build applications against your lakehouse. Without this connection, Cascade treats Dremio like a generic database. With it, the agent knows your schemas, business logic encoded in views, and the correct Dremio SQL dialect.

Windsurf's Cascade is especially well-suited for data projects because it can chain together discovery, querying, code generation, and testing in a single autonomous flow. Ask it to explore your Dremio catalog, identify relevant tables, write a pipeline, and generate tests — all in one prompt.

This post covers four approaches, ordered from quickest setup to most customizable.

Setting Up Windsurf

If you do not already have Windsurf installed:

Download Windsurf from windsurf.com (available for macOS, Linux, and Windows).
Install it by running the installer. Windsurf is a VS Code fork, so all VS Code extensions and themes work.
Sign in with a Windsurf account. The free tier includes limited Cascade credits; Pro provides expanded access.
Open a project by selecting File > Open Folder and pointing to your project directory.
Open Cascade by pressing Cmd+L (macOS) or Ctrl+L to access the agentic AI chat panel.

If you are migrating from VS Code or Cursor, your existing extensions and settings transfer automatically.

Approach 1: Connect the Dremio Cloud MCP Server

The Model Context Protocol (MCP) is an open standard that lets AI tools call external services. Every Dremio Cloud project ships with a built-in MCP server. Windsurf supports MCP natively through its Cascade settings.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup. For Windsurf, you configure the MCP connection through the Cascade settings or mcp_config.json.

Find Your Project's MCP Endpoint

Log into Dremio Cloud and open your project. Navigate to Project Settings > Info. The MCP server URL is listed on the project overview page. Copy it.

Set Up OAuth in Dremio Cloud

Go to Settings > Organization Settings > OAuth Applications.
Click Add Application and enter a name (e.g., "Windsurf MCP").
Add the appropriate redirect URIs.
Save and copy the Client ID.

Configure Windsurf's MCP Connection

You have two options:

Option A: Via Settings UI

Open Windsurf Settings and navigate to Cascade > MCP. Click Add custom server and paste your Dremio MCP configuration.

Option B: Via mcp_config.json

Create or edit ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "dremio": {
      "url": "https://YOUR_PROJECT_MCP_URL"
    }
  }
}

Restart Windsurf. Cascade now has access to Dremio's MCP tools:

GetUsefulSystemTableNames returns available tables with descriptions.
GetSchemaOfTable returns column names, types, and metadata.
GetDescriptionOfTableOrSchema pulls wiki descriptions from the catalog.
GetTableOrViewLineage shows upstream dependencies.
RunSqlQuery executes SQL and returns results as JSON.

Test the connection by opening Cascade and asking: "What tables are available in Dremio?" Cascade will call GetUsefulSystemTableNames and return your catalog contents.

Self-Hosted Alternative

For Dremio Software deployments, use the open-source dremio-mcp server:

git clone https://github.com/dremio/dremio-mcp
cd dremio-mcp
uv run dremio-mcp-server config create dremioai \
  --uri https://your-dremio-instance.com \
  --pat YOUR_PERSONAL_ACCESS_TOKEN

In mcp_config.json:

{
  "mcpServers": {
    "dremio": {
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

Approach 2: Use Windsurf Rules for Dremio Context

Windsurf supports .windsurfrules files in your project root for persistent AI instructions. These work similarly to .cursorrules and are loaded into every Cascade interaction.

Project-Wide Rules

Create a .windsurfrules file in your project root:

# Dremio SQL Conventions
- Use CREATE FOLDER IF NOT EXISTS (not CREATE NAMESPACE or CREATE SCHEMA)
- Tables in the Open Catalog use folder.subfolder.table_name without a catalog prefix
- External federated sources use source_name.schema.table_name
- Cast DATE to TIMESTAMP for consistent joins
- Use TIMESTAMPDIFF for duration calculations

# Credentials
- Never hardcode Personal Access Tokens. Use environment variable: DREMIO_PAT
- Dremio Cloud endpoint: environment variable DREMIO_URI

# Terminology
- Call it "Agentic Lakehouse", not "data warehouse"
- "Reflections" are pre-computed optimizations, not "materialized views"

Windsurf also reads .cursorrules as a fallback if no .windsurfrules file is present, so if your team uses Cursor alongside Windsurf, shared rules files work across both editors.

Cascade Memory and Context

Cascade has a persistent memory system. As you work with Dremio tables, Cascade remembers the schemas, query patterns, and conventions it has encountered. This means subsequent requests in the same project get more accurate over time without needing to re-read context files.

Approach 3: Install Pre-Built Dremio Skills and Docs

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill (Community)

The dremio-agent-skill repository contains a comprehensive skill directory with knowledge files and a .cursorrules file that Windsurf reads as a fallback.

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cd dremio-agent-skill
./install.sh

Choose Local Project Install (Copy) to copy the .cursorrules file and knowledge directory into your project. Windsurf will pick up the rules file automatically.

dremio-agent-md (Community)

The dremio-agent-md repository provides a master protocol file and browsable documentation sitemaps:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Reference it in your .windsurfrules:

For Dremio SQL validation, read DREMIO_AGENT.md in the dremio-agent-md directory.
Use the sitemaps in dremio_sitemaps/ to verify syntax before generating SQL.

Approach 4: Build Your Own Windsurf Rules

Create a custom .windsurfrules with your team's specific Dremio environment:

# Team Dremio Context

## Table Schemas (updated weekly)
- For table schemas, read ./docs/table-schemas.md
- For SQL conventions, read ./docs/dremio-conventions.md
- For common queries, read ./docs/common-queries.md

## Naming Standards
- Bronze: raw.*, Silver: cleaned.*, Gold: analytics.*
- Always use TIMESTAMP, never DATE
- Validate function names against docs/dremio-conventions.md

Export your actual schemas from Dremio and keep them updated. Cascade's memory system means it learns your patterns over time, but explicit rules ensure consistency from the first interaction.

Using Dremio with Windsurf: Practical Use Cases

Once Dremio is connected, Cascade can execute complex multi-step data projects autonomously. Here are detailed examples.

Ask Natural Language Questions About Your Data

Open Cascade and ask questions in plain English:

"What were our top 10 products by revenue last quarter? Break it down by region and show the growth rate compared to the previous quarter."

Cascade uses MCP to discover your tables, writes the SQL, runs it against Dremio, and returns formatted results. Its multi-step nature means it can chain multiple queries together autonomously.

Follow up immediately:

"For products with declining growth, pull the customer reviews and support tickets. Is there a pattern between product issues and revenue decline?"

Cascade maintains full context and chains together cross-table queries without prompting. This turns the editor into a data analysis workstation.

Build a Locally Running Dashboard

Use Cascade for multi-step project generation:

"Query our gold-layer sales views in Dremio and build a local HTML dashboard with Chart.js. Include monthly revenue trends, top products by region, and customer metrics. Add date range filters, a dark theme, and export buttons. Create separate HTML, CSS, and JavaScript files."

Cascade will autonomously:

Call MCP to discover gold-layer views and schemas
Write and execute SQL queries for each metric
Create index.html with the dashboard layout
Create styles.css with dark theme and responsive design
Create dashboard.js with Chart.js configurations
Wire everything together and save to your project

Open index.html in a browser for a complete interactive dashboard. Cascade's agentic flow handles the entire process without manual intervention.

Create a Data Exploration App

Build interactive tools in one prompt:

"Create a Streamlit app connected to Dremio via dremioframe. Include a schema browser with table previews, a SQL editor with syntax highlighting, CSV download, and charting for numeric columns. Generate all the files including requirements.txt and a README."

Cascade generates the full application stack and wires the components together. Run streamlit run app.py for a local data explorer.

Generate Data Pipeline Scripts

Automate data engineering:

"Write a Medallion Architecture pipeline using dremioframe. Bronze: ingest raw events from S3. Silver: deduplicate, validate required fields, cast timestamps. Gold: aggregate daily metrics and build customer lifetime value calculations. Include structured logging, retry logic, and dry-run mode."

Cascade writes the pipeline code, creates test files, and can execute a dry run to verify the logic against your live Dremio instance.

Build API Endpoints Over Dremio Data

Create backend services:

"Build a FastAPI app that queries Dremio gold-layer views via dremioframe. Add endpoints for customer segments, revenue analytics, and cohort retention. Include Pydantic models, caching, and OpenAPI docs."

Cascade generates the complete API project. Run uvicorn main:app --reload for a local API connected to your lakehouse.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time access
Windsurf Rules	10 minutes	Convention enforcement, persistent AI instructions	Teams with specific SQL standards
Pre-Built Skills	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Quick start with broad coverage
Custom Rules	30+ minutes	Tailored schemas, patterns, and team conventions	Mature teams with project-specific needs

Start with the MCP server for immediate value. Add a .windsurfrules file for Dremio conventions. Let Cascade's memory build on your patterns over time.

Get Started

Sign up for a free Dremio Cloud trial (30 days, $400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Add it in Windsurf's Cascade > MCP settings or mcp_config.json.
Clone dremio-agent-skill and run ./install.sh.
Open Cascade and ask it to explore your Dremio catalog.

Dremio's Agentic Lakehouse gives Cascade accurate data context, and Cascade's multi-step autonomous flows turn that context into complete data projects.

For more on the Dremio MCP Server, check out the official documentation or enroll in the free Dremio MCP Server course on Dremio University.

How to Use Dremio with OpenWork: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 19:00:00 GMT

OpenWork is an open-source desktop AI agent built on the OpenCode engine. It runs entirely on your machine with your own API keys, giving you full control over your data and your AI costs. Dremio is a unified lakehouse platform built on open standards like Apache Iceberg, Apache Arrow, and Apache Polaris.

Both tools share a local-first philosophy. Dremio stores data in open formats with no vendor lock-in. OpenWork runs on your hardware with no cloud dependency for the agent itself. Connecting them creates an open-source analytics stack where your coding agent queries your lakehouse without sending data through third-party services.

OpenWork inherits OpenCode's AGENTS.md support, opencode.json configuration, and MCP integration. If you have already written Dremio context files for OpenCode or OpenAI Codex, they work in OpenWork without modification. The desktop application adds a graphical interface, integrated file browser, and agent chat panel on top of the terminal experience.

The local-first model has specific advantages for data work. Your Dremio queries and results stay on your machine. Your API keys are stored locally. The agent code runs in your environment. For teams that handle sensitive data or operate under compliance constraints, this architecture keeps the AI agent within your security perimeter.

This post covers four approaches, from a five-minute MCP connection to a fully custom Dremio configuration.

Setting Up OpenWork

If you do not already have OpenWork installed:

Download OpenWork from openwork.software (available for macOS, Linux, and Windows).
Install it by following the platform-specific instructions.
Configure your AI model by adding your API key (OpenAI, Anthropic, or another supported provider) in the application settings.
Open a project by selecting your project directory in the OpenWork file browser.

OpenWork is built on the OpenCode engine but provides a desktop GUI with an integrated file browser, agent chat panel, and visual output display. It runs entirely on your machine with your own API keys, giving you full control over costs and data privacy.

Approach 1: Connect the Dremio Cloud MCP Server

Every Dremio Cloud project includes a built-in MCP server. OpenWork supports MCP through its inherited opencode.json configuration.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup. For OpenWork, you configure the MCP connection through opencode.json:

Find Your MCP Endpoint and Set Up OAuth

Log into Dremio Cloud and find your MCP URL under Project Settings > Info.
Go to Settings > Organization Settings > OAuth Applications.
Create a new application with an appropriate redirect URI.
Copy the Client ID.

Configure OpenWork's MCP Connection

Add the Dremio server to your opencode.json:

{
  "mcp": {
    "dremio": {
      "url": "https://YOUR_PROJECT_MCP_URL",
      "auth": {
        "type": "oauth",
        "clientId": "YOUR_CLIENT_ID"
      }
    }
  }
}

Place this at your project root or globally at ~/.config/opencode/opencode.json. After configuration, OpenWork can call Dremio's MCP tools:

GetUsefulSystemTableNames returns available tables with descriptions.
GetSchemaOfTable returns column details and metadata.
GetDescriptionOfTableOrSchema pulls wiki descriptions from the catalog.
RunSqlQuery executes SQL and returns JSON results.

Self-Hosted MCP

For Dremio Software deployments, use the open-source dremio-mcp server:

git clone https://github.com/dremio/dremio-mcp
cd dremio-mcp
uv run dremio-mcp-server config create dremioai \
  --uri https://your-dremio-instance.com \
  --pat YOUR_PERSONAL_ACCESS_TOKEN

Configure OpenWork to run the local server:

{
  "mcp": {
    "dremio": {
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

The server supports three modes: FOR_DATA_PATTERNS (data exploration, default), FOR_SELF (system introspection for diagnosing performance), and FOR_PROMETHEUS (metrics correlation). The local-first nature of OpenWork pairs well with the self-hosted MCP option, as both components run entirely on your infrastructure.

Approach 2: Use AGENTS.md for Dremio Context

OpenWork inherits AGENTS.md support from OpenCode. The same file works in OpenWork, OpenCode, and OpenAI Codex.

Writing a Dremio-Focused AGENTS.md

Create AGENTS.md in your project root:

# Agent Configuration

## Dremio Lakehouse

This project uses Dremio Cloud as its lakehouse.

### SQL Conventions
- Use `CREATE FOLDER IF NOT EXISTS` (not CREATE NAMESPACE)
- Open Catalog tables: `folder.subfolder.table_name` (no catalog prefix)
- External sources: `source_name.schema.table_name`
- Cast DATE to TIMESTAMP for join consistency
- Use TIMESTAMPDIFF for duration calculations

### Credentials
- PAT: env var `DREMIO_PAT`
- Endpoint: env var `DREMIO_URI`
- Never hardcode credentials

### References
- SQL reference: https://docs.dremio.com/current/reference/sql/
- REST API: https://docs.dremio.com/current/reference/api/
- Local SQL docs: ./docs/dremio-sql-reference.md

### Terminology
- "Agentic Lakehouse" not "data warehouse"
- "Reflections" not "materialized views"
- "Open Catalog" built on Apache Polaris

OpenWork auto-scans this file at project start. Global defaults go in ~/.config/opencode/AGENTS.md and project-level files override them.

Cross-Tool Portability

The AGENTS.md you write for OpenWork works identically in OpenCode and OpenAI Codex. If your team uses multiple tools, you maintain one Dremio configuration file instead of separate context files for each tool.

Approach 3: Install Pre-Built Dremio Skills and Docs

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill (Community)

The dremio-agent-skill repository provides a complete skill directory with SKILL.md, knowledge files (CLI, Python SDK, SQL, REST API), and AGENTS.md in the rules/ directory.

For OpenWork, copy the AGENTS.md:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cp dremio-agent-skill/dremio-skill/rules/AGENTS.md ./AGENTS.md

Or run the full installer for broader integration:

cd dremio-agent-skill && ./install.sh

dremio-agent-md (Community)

The dremio-agent-md repository provides DREMIO_AGENT.md and documentation sitemaps. Clone it alongside your project:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Tell OpenWork: "Read DREMIO_AGENT.md in the dremio-agent-md directory and use the sitemaps to validate SQL." OpenWork's desktop interface makes it easy to have the agent-md folder open in the file browser while working on your project.

Approach 4: Build a Custom Dremio Configuration

Custom AGENTS.md with Knowledge Files

Create a project structure with reference docs:

project-root/
  AGENTS.md
  docs/
    dremio-sql-reference.md
    team-schemas.md
    dremioframe-patterns.md

Reference the docs in your AGENTS.md so OpenWork reads them on demand. Populate with your actual table schemas exported from Dremio, team-specific SQL patterns, and dremioframe code snippets.

Custom Agents

OpenWork inherits OpenCode's custom agent system. Create dedicated Dremio agents in .opencode/agents/:

# .opencode/agents/dremio-analyst.md
---
description: Dremio data analyst agent
mode: subagent
---

You are a data analyst working with Dremio Cloud.
1. Use the MCP connection to explore tables
2. Follow Dremio SQL conventions (CREATE FOLDER IF NOT EXISTS, etc.)
3. Validate function names against the SQL reference
4. Never hardcode credentials

This subagent uses a separate model and context window dedicated to Dremio tasks, producing higher-quality SQL than a general-purpose agent.

Using Dremio with OpenWork: Practical Use Cases

Once Dremio is connected, OpenWork can generate complete data applications. Here are detailed examples.

Ask Natural Language Questions About Your Data

Type a question in the OpenWork chat panel and get answers from your lakehouse:

"What is the average order value by customer segment for Q4? Which segment grew the fastest compared to Q3?"

OpenWork queries Dremio through MCP, computes the comparison, and returns a formatted answer with the SQL it ran. This turns your desktop agent into a local, private data analyst that works with production data.

Follow up with deeper analysis:

"For the fastest-growing segment, show the top 10 customers by order frequency. Are they new customers or returning? Pull their first order date and total lifetime value."

OpenWork maintains context from the previous question and writes progressively more complex queries. Because everything runs locally, your data never leaves your machine.

This pattern is especially powerful for teams with data sovereignty requirements. The AI model processes your prompt, but the data stays on your infrastructure.

Build a Locally Running Dashboard

Ask OpenWork to create a self-contained dashboard:

"Query our gold-layer views in Dremio for monthly revenue, active users, and churn rate over the last 12 months. Build an HTML dashboard with Plotly.js charts. Include filters for region and product line. Add a dark theme and export-to-PNG buttons."

OpenWork will:

Use MCP to discover gold-layer views and their schemas
Write and execute SQL queries for each metric
Generate an HTML file with Plotly.js interactive charts
Add dropdown filters for region and product line
Include export functionality and responsive layout
Save everything to your project folder

Open it in a browser for a fully interactive dashboard running from a local file. No server required. The Plotly.js charts support zoom, pan, and hover tooltips.

Create a Data Exploration App

Build a more sophisticated tool:

"Create a Streamlit app that connects to Dremio using dremioframe. Add a sidebar for selecting schemas and tables, a schema viewer, a data preview with pagination, a custom SQL query editor with results displayed as a table, and CSV download buttons."

OpenWork writes the full Python application:

app.py with Streamlit layout, dremioframe connection, and query execution
requirements.txt with pinned dependencies
.env.example with required environment variables
README.md with setup instructions

Run streamlit run app.py and you have a local data exploration tool connected to your lakehouse. Since both OpenWork and the app run on your machine, your data never leaves your infrastructure.

Generate Data Pipeline Scripts

Automate your ETL workflows:

"Write a Python script using dremioframe that reads raw CSV data from S3, creates a bronze table in Dremio, builds silver views with data quality rules (null checks, type validation, deduplication), and creates a gold view with business logic aggregations. Include error handling, logging, and a dry-run mode."

OpenWork uses the Dremio skill knowledge to write pipeline code that follows your team's Medallion Architecture conventions. The script includes structured logging, retry logic, and a summary report at the end.

Build API Endpoints Over Dremio Data

Serve lakehouse data to other applications:

"Build a FastAPI application that connects to Dremio using dremioframe. Create endpoints for device metrics, alert summaries, and historical trends. Include request validation, response caching with a 5-minute TTL, and auto-generated API docs."

OpenWork generates the complete server with proper error handling and connection management. Run uvicorn main:app --reload for a local API connected to your lakehouse.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog	NL data exploration, building apps
AGENTS.md	10 minutes	Convention enforcement, cross-tool portable	Multi-tool teams
Pre-Built Skills	5 minutes	Broad Dremio knowledge	Quick start
Custom Config	30+ minutes	Tailored agents, schemas, patterns	Advanced multi-agent workflows

OpenWork's advantage is the local-first model. Your agent, your API keys, your data connections all run on your machine. Combined with Dremio's open lakehouse formats, you get a fully controlled analytics stack.

Start with the MCP server for immediate access to your data. Layer in AGENTS.md for conventions and custom agents for specialized Dremio workflows. If your team already uses OpenCode or Codex, your existing AGENTS.md and MCP configuration work in OpenWork immediately.

The local-first model means you can evaluate OpenWork with Dremio without any organizational approval process. Install it on your machine, connect it to your Dremio Cloud project, and start querying. If it works for you, share the AGENTS.md and opencode.json files with your team so they can replicate the same setup on their machines.

Get Started

Sign up for Dremio Cloud free for 30 days ($400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Add it to your opencode.json.
Clone dremio-agent-skill and copy the AGENTS.md.
Ask OpenWork to explore your catalog and build a local dashboard from your data.

Dremio's Agentic Lakehouse provides what OpenWork needs: the semantic layer for business context, query federation for universal data access, and Reflections for interactive speed. Both platforms embrace open standards and local-first operation, making them a natural fit for teams that prioritize data sovereignty and transparency.

For more on the Dremio MCP Server, see the official documentation or enroll in the free Dremio MCP Server course on Dremio University.

How to Use Dremio with OpenCode: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 18:00:00 GMT

OpenCode is an open-source, terminal-based AI coding agent released under the MIT license. It provides a TUI with split panes, uses the Language Server Protocol (LSP) for deep codebase understanding, and maintains persistent project context through file-based memory. Dremio is a unified lakehouse platform built on open standards like Apache Iceberg, Apache Arrow, and Apache Polaris.

The open-source philosophy aligns. Dremio stores data in open formats with no vendor lock-in. OpenCode gives you full control over your AI coding agent with no proprietary restrictions. Connecting them means your open-source agent can query an open lakehouse, validate SQL against real schemas, and generate scripts using your team's actual conventions.

OpenCode uses the same AGENTS.md standard as OpenAI Codex, so the Dremio context files you write work across both tools. It also supports custom agents with dedicated prompts and model configurations, which opens up a Dremio-specific agent pattern that other tools do not offer. You can create a dedicated data analyst subagent that uses a reasoning model for SQL generation while your primary agent uses a faster model for application code.

OpenCode's LSP integration gives it another advantage. The agent analyzes imports, dependencies, and file structure at the language level. When you combine this with Dremio's MCP server, the agent understands both your code structure and your data structure simultaneously.

This post covers four approaches, ordered from quickest to most customizable.

Setting Up OpenCode

If you do not already have OpenCode installed:

Install Go (version 1.23 or later) from go.dev.
Install OpenCode:
```
go install github.com/opencode-ai/opencode@latest
```
Or use Homebrew: brew install opencode.
Configure your AI model by setting the OPENAI_API_KEY, ANTHROPIC_API_KEY, or other model provider key in your environment.
Launch OpenCode by running opencode in your terminal from any project directory.

OpenCode provides a TUI with split panes, LSP-powered code understanding, and a multi-agent architecture that lets you define specialized subagents for different tasks. It is open-source under the MIT license.

Approach 1: Connect the Dremio Cloud MCP Server

Every Dremio Cloud project includes a built-in MCP server. OpenCode supports MCP natively through its opencode.json configuration.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup. For OpenCode, you configure the MCP connection through opencode.json:

Find Your Project's MCP Endpoint

Log into Dremio Cloud and open your project. Go to Project Settings > Info and copy the MCP server URL.

Set Up OAuth in Dremio Cloud

Navigate to Settings > Organization Settings > OAuth Applications.
Click Add Application and name it (e.g., "OpenCode MCP").
Add the redirect URI for your setup.
Copy the Client ID.

Configure OpenCode's MCP Connection

Add the Dremio MCP server to your opencode.json:

{
  "mcp": {
    "dremio": {
      "url": "https://YOUR_PROJECT_MCP_URL",
      "auth": {
        "type": "oauth",
        "clientId": "YOUR_CLIENT_ID"
      }
    }
  }
}

For global configuration, place it in ~/.config/opencode/opencode.json. For project-specific config, place it at the project root.

After configuring, OpenCode can call Dremio's MCP tools:

GetUsefulSystemTableNames lists tables with descriptions.
GetSchemaOfTable returns column names, types, and metadata.
GetDescriptionOfTableOrSchema pulls wiki descriptions and labels from the catalog.
GetTableOrViewLineage shows data lineage.
RunSqlQuery executes SQL and returns JSON results.

Self-Hosted Alternative

For Dremio Software, use the open-source dremio-mcp server:

git clone https://github.com/dremio/dremio-mcp
cd dremio-mcp
uv run dremio-mcp-server config create dremioai \
  --uri https://your-dremio-instance.com \
  --pat YOUR_PERSONAL_ACCESS_TOKEN

Then configure OpenCode to run the local server in opencode.json:

{
  "mcp": {
    "dremio": {
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

The server supports FOR_DATA_PATTERNS (query and explore), FOR_SELF (system introspection), and FOR_PROMETHEUS (metrics). Most coding workflows use FOR_DATA_PATTERNS, the default mode. It gives the agent full access to explore your catalog, read schemas, pull wiki descriptions, and run SQL queries.

If your team also handles Dremio administration, FOR_SELF mode lets the agent analyze job history, resource utilization, and query performance. This is useful for platform engineering tasks where you need the agent to diagnose slow queries or suggest Reflection configurations. FOR_PROMETHEUS connects to your monitoring stack for correlating Dremio metrics with broader system observability.

For Dremio Cloud users, the hosted MCP server is the simpler option. No local installation, OAuth-based auth, and your existing access controls apply automatically. The self-hosted server gives more control and works with on-premise Dremio Software deployments.

Approach 2: Use AGENTS.md for Dremio Context

OpenCode shares the AGENTS.md standard with OpenAI Codex. It auto-scans for this file at project start and uses it to guide agent behavior.

AGENTS.md Placement

Project root: AGENTS.md applies to the current project.
Global: ~/.config/opencode/AGENTS.md applies across all projects.
Project-level files override global defaults.

Writing a Dremio-Focused AGENTS.md

# Agent Configuration

## Dremio Lakehouse

This project uses Dremio Cloud as its lakehouse.

### SQL Conventions
- Use `CREATE FOLDER IF NOT EXISTS` for namespace creation
- Open Catalog tables: `folder.subfolder.table_name` (no catalog prefix)
- External sources: `source_name.schema.table_name`
- Cast DATE to TIMESTAMP for join consistency
- Use TIMESTAMPDIFF for duration calculations

### Credentials
- PAT: env var `DREMIO_PAT`
- Endpoint: env var `DREMIO_URI`
- Never hardcode credentials

### References
- SQL syntax: https://docs.dremio.com/current/reference/sql/
- REST API: https://docs.dremio.com/current/reference/api/
- Local SQL reference: ./docs/dremio-sql-reference.md

### Terminology
- "Agentic Lakehouse" not "data warehouse"
- "Reflections" not "materialized views"
- "Open Catalog" built on Apache Polaris

Run /init inside OpenCode to generate a starter AGENTS.md from a project scan, then add the Dremio sections above.

Custom Agents for Dremio-Specific Workflows

OpenCode supports defining custom agents in .opencode/agents/. This is a capability that most other tools lack. You can create a dedicated Dremio agent with its own system prompt, model choice, and tool permissions.

Create .opencode/agents/dremio-analyst.md:

---
description: Dremio data analyst agent
mode: subagent
---

You are a data analyst working with Dremio Cloud. Your job is to:
1. Explore available tables using the MCP connection
2. Write SQL queries that follow Dremio conventions
3. Use TIMESTAMPDIFF, not DATEDIFF
4. Use CREATE FOLDER IF NOT EXISTS, not CREATE SCHEMA
5. Always validate function names against the SQL reference before using them
6. Never hardcode credentials; use environment variables

This agent runs as a subagent that the primary agent can invoke for Dremio-specific tasks. You can configure it with a different model (for example, a reasoning model optimized for SQL generation) and restrict its tool access to only the Dremio MCP server.

Approach 3: Install Pre-Built Dremio Skills and Docs

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill (Community)

The dremio-agent-skill repository contains a complete agent skill with SKILL.md, knowledge files, and an AGENTS.md in the rules/ directory.

For OpenCode, copy the AGENTS.md from the skill to your project:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cp dremio-agent-skill/dremio-skill/rules/AGENTS.md ./AGENTS.md

Or run the full installer for broader integration:

cd dremio-agent-skill
./install.sh

The skill includes knowledge files covering Dremio CLI, Python SDK (dremioframe), SQL syntax, and REST API endpoints.

dremio-agent-md (Community)

The dremio-agent-md repository provides a DREMIO_AGENT.md protocol file and hierarchical documentation sitemaps.

Clone it and tell OpenCode to read the protocol:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Instruct OpenCode: "Read DREMIO_AGENT.md in the dremio-agent-md directory. Use the sitemaps to verify SQL syntax before generating code."

This is especially powerful with OpenCode's LSP-based context engine. The agent can cross-reference the Dremio sitemaps with your actual project imports and file structure, ensuring that the SQL it generates fits both the Dremio dialect and your project conventions.

Approach 4: Build a Custom Dremio Agent

OpenCode's custom agent system is its differentiator for Dremio integration. While other tools limit you to context files, OpenCode lets you define a purpose-built Dremio agent.

Multi-Agent Architecture

Create a primary coding agent plus a Dremio-focused subagent:

.opencode/agents/
  dremio-analyst.md       # Subagent for SQL and data queries
  dremio-pipeline.md      # Subagent for ETL/pipeline scripts

Each agent gets its own system prompt, model configuration, and tool permissions. The primary agent delegates Dremio tasks to the appropriate subagent, which has the full Dremio context loaded while keeping the primary agent's context window focused on application code.

This separation matters for large projects. A data pipeline subagent can be configured with a reasoning-capable model (like a Chain of Thought model) that excels at complex SQL generation, while your primary coding agent uses a faster model for application logic. The Dremio subagent's tool permissions can be restricted to only the Dremio MCP server, preventing it from accidentally modifying application files.

Knowledge Files

Pair your custom agents with reference documentation:

docs/
  dremio-sql-reference.md
  team-schemas.md
  dremioframe-patterns.md

Reference these in both your AGENTS.md and your custom agent prompts. OpenCode's file-based memory system ensures the agent retains context from these references across interactions. Export your actual table schemas from Dremio's catalog and save them as markdown. Include dremioframe code snippets for common operations like querying, creating views, and managing branches. Add REST API call patterns for your CI/CD pipelines.

Using Dremio with OpenCode: Practical Use Cases

Once Dremio is connected, OpenCode's multi-agent architecture enables sophisticated data workflows. Here are detailed examples.

Ask Natural Language Questions About Your Data

Type a question in OpenCode's TUI and get answers from your lakehouse:

"Which product categories have the highest return rates? Cross-reference with customer satisfaction scores and identify correlations."

OpenCode routes this to the Dremio subagent, which discovers the relevant tables via MCP, writes a multi-table join with aggregations, runs it against Dremio, and returns analysis with the underlying SQL. The primary agent stays focused on your code context while the Dremio subagent handles the data work.

Dig deeper with follow-up analysis:

"For the categories with highest returns, pull the top reasons from the returns table. Group by product SKU and show which specific items are driving the category-level numbers."

The Dremio subagent already knows the schema context from the previous query and generates the follow-up efficiently.

Build a Locally Running Dashboard

Ask OpenCode to create a visualization:

"Query Dremio's gold-layer views for inventory levels, reorder rates, and supplier lead times. Build a local HTML dashboard with ECharts showing stock trends, a forecasting chart, and supplier performance scorecards. Include a responsive layout and dark theme."

OpenCode's multi-agent system handles this: the Dremio subagent writes and executes the SQL queries, while the primary agent generates the HTML/CSS/JavaScript:

Dremio subagent discovers inventory views and pulls data
Primary agent generates the HTML structure with ECharts
Data is embedded as JSON in the generated file
Interactive filters for warehouse, category, and date range
Responsive layout that works on desktop and tablet

Open it in a browser for an interactive supply chain dashboard running from a local file.

Create a Data Exploration App

Build a full application leveraging the multi-agent architecture:

"Create a Python Dash app that uses dremioframe to connect to Dremio. Include a catalog browser, table schema viewer with column statistics, and a multi-tab interface for SQL queries, data profiling, and anomaly detection. Add a connection settings page."

OpenCode delegates the work: the Dremio subagent writes the dremioframe connection code and SQL queries, while the primary agent builds the Dash UI components and layout:

Multi-tab interface with catalog browser, schema viewer, SQL editor, and profiler
Column statistics calculated from Dremio metadata
Anomaly detection using basic IQR analysis on numeric columns
Connection settings stored in .env with a settings page for updates

Run python app.py and your team has a local data platform connected to the lakehouse.

Generate Pipeline Scripts with Agent Collaboration

Automate data engineering with Dremio-aware code:

"Using the Dremio skill, create a Python ETL pipeline that processes IoT sensor data. Create bronze tables for raw readings, silver views that apply calibration offsets and flag anomalies (readings outside 3 standard deviations), and gold views that aggregate by device and time window. Include retry logic and structured logging."

The Dremio subagent writes the pipeline code using correct Dremio SQL conventions and bronze-silver-gold patterns, while the primary agent handles file management, error handling, and test generation. The result is production-quality code with proper separation of concerns.

Build API Endpoints Over Dremio Data

Serve lakehouse data to downstream applications:

"Build a FastAPI service with endpoints for IoT device metrics, alert summaries, and historical trends. Connect to Dremio using dremioframe. Add WebSocket support for real-time data streaming and Pydantic response models."

OpenCode generates the full API server with the Dremio subagent handling query logic and the primary agent building the FastAPI framework.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog	Data analysis, real-time access
AGENTS.md	10 minutes	Convention enforcement, portable config	Cross-tool consistency
Pre-Built Skills	5 minutes	Broad Dremio knowledge	Quick start
Custom Agent	30+ minutes	Dedicated Dremio subagent with own model/prompt	Advanced multi-agent workflows

OpenCode's custom agent system makes the fourth approach more powerful than in other tools. A dedicated Dremio subagent with its own reasoning model and restricted tool access produces higher-quality SQL than a general-purpose agent trying to handle both application code and data queries in the same context.

Combine the MCP server for live data access with a custom Dremio agent for SQL generation, and an AGENTS.md for project-wide conventions. This three-layer stack gives you the strongest Dremio integration available in any open-source coding tool.

If you are coming from Claude Code or Codex and want an open-source alternative, start with the AGENTS.md approach since your existing file works directly in OpenCode. Add the MCP connection for live data, then explore custom agents to see if the multi-agent architecture improves your workflow.

Get Started

Sign up for Dremio Cloud free for 30 days ($400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Add it to your opencode.json.
Clone dremio-agent-skill and copy the AGENTS.md.
Start OpenCode and ask it to explore your Dremio catalog.

Dremio's Agentic Lakehouse provides what OpenCode's agents need for accurate analytics: the semantic layer delivers business context, query federation delivers universal data access, and Reflections deliver interactive speed. Both platforms are built on open standards, and connecting them gives you an open-source analytics stack from agent to lakehouse.

For more on the Dremio MCP Server, see the official documentation or take the free Dremio MCP Server course on Dremio University.

How to Use Dremio with OpenAI Codex CLI: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 17:00:00 GMT

OpenAI Codex CLI is a terminal-based coding agent built in Rust. It reads your codebase, writes files, executes commands, and supports MCP for connecting to external data services. Dremio is a unified lakehouse platform that provides the business context, universal data access, and query speed that coding agents need to produce accurate, working analytics code.

Codex uses AGENTS.md as its primary context file. This is an open standard designed to work across multiple AI tools, so the Dremio configuration you write for Codex also works with other AGENTS.md-compatible tools. That portability matters if your team uses different agents.

Without a Dremio connection, Codex treats your lakehouse like any generic database. It may guess at table names, hallucinate SQL functions, or ignore your team's naming conventions. With a proper connection, Codex knows your schema, your business logic encoded in virtual views, and the right Dremio SQL dialect.

Codex's support for the AGENTS.md open standard is worth highlighting. Unlike tool-specific context files, AGENTS.md works across multiple AI agents. Write it once for Codex and your team members using OpenCode, OpenWork, or any other AGENTS.md-compatible tool get the same context without maintaining separate files.

This post covers four approaches, ordered from quickest setup to most customizable. Start with the one that matches your current needs, and layer in the others as your Dremio usage grows.

Setting Up OpenAI Codex CLI

If you do not already have Codex CLI installed:

Install Node.js (version 22 or later) from nodejs.org.
Install Codex globally via npm:
```
npm install -g @openai/codex
```
Launch Codex by running codex in your terminal from any project directory.
Authenticate with your OpenAI API key. Codex uses the OPENAI_API_KEY environment variable.

Codex runs in your terminal and reads your project files for context. It supports three autonomy modes: suggest (proposes changes), auto-edit (applies file edits), and full-auto (runs commands without confirmation).

Approach 1: Connect the Dremio Cloud MCP Server

The Model Context Protocol (MCP) is an open standard that lets AI tools call external services. Every Dremio Cloud project ships with a built-in MCP server. Codex supports MCP natively, making this the fastest way to give the agent direct access to your data.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup including /dremio-setup for step-by-step configuration. For Codex, you configure the MCP connection through your project settings:

Find Your Project's MCP Endpoint

Log into Dremio Cloud and open your project. Navigate to Project Settings > Info. Copy the MCP server URL from the project overview page.

Set Up OAuth in Dremio Cloud

Dremio's hosted MCP server uses OAuth for authentication. Your existing access controls apply to every query Codex runs.

Go to Settings > Organization Settings > OAuth Applications.
Click Add Application and name it (e.g., "Codex MCP").
Add the redirect URI specific to your Codex client setup.
Save and copy the Client ID.

Configure Codex's MCP Connection

Codex reads MCP configuration from its settings. Add the Dremio server to your MCP configuration:

{
  "mcpServers": {
    "dremio": {
      "url": "https://YOUR_PROJECT_MCP_URL",
      "auth": {
        "type": "oauth",
        "clientId": "YOUR_CLIENT_ID"
      }
    }
  }
}

After configuring, Codex can call Dremio's MCP tools directly:

GetUsefulSystemTableNames lists available tables with descriptions.
GetSchemaOfTable returns column names, types, and metadata.
GetDescriptionOfTableOrSchema pulls wiki descriptions and labels from the catalog.
GetTableOrViewLineage shows upstream data dependencies.
RunSqlQuery executes SQL and returns results as JSON.

Test it by asking Codex: "What tables are available in Dremio?" The agent will call the appropriate MCP resource and return your catalog contents.

Self-Hosted Alternative

For Dremio Software deployments, use the open-source dremio-mcp server:

git clone https://github.com/dremio/dremio-mcp
cd dremio-mcp
uv run dremio-mcp-server config create dremioai \
  --uri https://your-dremio-instance.com \
  --pat YOUR_PERSONAL_ACCESS_TOKEN

Then configure Codex to run the local server:

{
  "mcpServers": {
    "dremio": {
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

The self-hosted server supports three modes: FOR_DATA_PATTERNS for data exploration (default), FOR_SELF for system introspection, and FOR_PROMETHEUS for metrics correlation.

The FOR_DATA_PATTERNS mode is what you want for most coding workflows. It enables the agent to explore your catalog, read table schemas, pull wiki descriptions, and run SQL queries. The FOR_SELF mode is useful for DevOps and platform engineering tasks where you need the agent to analyze Dremio's own performance metrics, job history, and resource utilization. The FOR_PROMETHEUS mode connects to your monitoring stack for correlating Dremio-specific metrics with broader system observability.

For Dremio Cloud users, the hosted MCP server is the simpler choice. It requires no local installation, handles authentication through OAuth, and inherits your existing access controls. The self-hosted option gives you more control and works with Dremio Software deployments that are not in the cloud.

Approach 2: Use AGENTS.md for Dremio Context

AGENTS.md is an open standard for providing AI coding agents with project context. Codex auto-scans for this file at the start of every task. It defines your project structure, coding conventions, and tool-specific instructions.

How AGENTS.md Works in Codex

Codex supports layered guidance:

Global defaults: ~/.codex/AGENTS.md applies to every project.
Project-level: AGENTS.md at the repo root overrides global defaults.
Nested overrides: AGENTS.override.md in subdirectories provides directory-specific rules that take precedence over broader ones.

This layering is useful for monorepos where different subdirectories interact with Dremio differently.

Writing a Dremio-Focused AGENTS.md

Create AGENTS.md in your project root:

# Project Agent Configuration

## Dremio Lakehouse

This project uses Dremio Cloud as its lakehouse platform.

### SQL Conventions
- Use `CREATE FOLDER IF NOT EXISTS` for namespace creation
- Tables in the Open Catalog: `folder.subfolder.table_name` (no catalog prefix)
- External sources: `source_name.schema.table_name`
- Cast DATE columns to TIMESTAMP for consistent joins
- Use TIMESTAMPDIFF for duration calculations

### Credentials
- Personal Access Token: use env var `DREMIO_PAT`
- Cloud endpoint: use env var `DREMIO_URI`
- Never hardcode credentials in scripts

### Documentation References
- Dremio SQL reference: https://docs.dremio.com/current/reference/sql/
- REST API: https://docs.dremio.com/current/reference/api/
- For detailed SQL validation, read ./docs/dremio-sql-reference.md

### Terminology
- Use "Agentic Lakehouse" not "data warehouse"
- "Reflections" not "materialized views"
- "Open Catalog" is built on Apache Polaris

You can also run codex init to let Codex scan your project and scaffold an initial AGENTS.md. Then edit it to add the Dremio-specific sections shown above.

Nested Overrides for Multi-Schema Projects

If different directories in your project target different Dremio namespaces, use AGENTS.override.md:

# data-pipeline/AGENTS.override.md

## Dremio Namespace Override
All tables in this directory use the `etl_pipeline` top-level namespace.
Bronze views: etl_pipeline.bronze.*
Silver views: etl_pipeline.silver.*
Gold views: etl_pipeline.gold.*

This override applies only when Codex is working on files within the data-pipeline/ directory.

Portability Across Tools

One key advantage of AGENTS.md over tool-specific formats: the same file works with OpenCode, OpenWork, and any future tool that adopts the standard. Write it once for Codex and your team members using other AGENTS.md-compatible tools get the same Dremio context without extra setup.

This portability is especially valuable for teams that are still evaluating which AI coding tool to standardize on. Rather than committing to CLAUDE.md (Claude-only) or SKILL.md (Antigravity-optimized), AGENTS.md gives you a tool-agnostic foundation that carries your Dremio conventions forward regardless of which agent your team picks.

Approach 3: Install Pre-Built Dremio Skills and Docs

Two community-supported open-source repositories provide ready-made Dremio context for coding agents.

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill: Full Agent Skill (Community)

The dremio-agent-skill repository contains a comprehensive skill directory that teaches AI assistants how to interact with Dremio. It includes knowledge files for the CLI, Python SDK (dremioframe), SQL syntax, and REST API.

For Codex, the skill's rules/ directory includes an AGENTS.md file you can copy to your project root:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cp dremio-agent-skill/dremio-skill/rules/AGENTS.md ./AGENTS.md

This gives Codex the Dremio conventions and references without running the full skill installer. For broader integration, run the installer:

cd dremio-agent-skill
./install.sh

Choose Local Project Install (Copy) to copy the skill directory into your project, or Global Install (Symlink) for system-wide access.

dremio-agent-md: Documentation Protocol (Community)

The dremio-agent-md repository provides a DREMIO_AGENT.md master protocol file and a browsable sitemap of the Dremio documentation.

Clone it alongside your project:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Then tell Codex to read the protocol file: "Read DREMIO_AGENT.md in the dremio-agent-md directory. Use the sitemaps in dremio_sitemaps/ to verify Dremio syntax before generating SQL."

This is especially useful for SQL validation. The agent navigates the sitemaps to find correct function signatures and reserved words instead of relying on training data.

Approach 4: Build Your Own Dremio Agent Configuration

If the pre-built options do not match your workflow, create a custom configuration.

Custom AGENTS.md with Knowledge Files

Create a directory structure that pairs your AGENTS.md with reference documents:

project-root/
  AGENTS.md
  docs/
    dremio-sql-reference.md
    team-schemas.md
    dremioframe-patterns.md

In your AGENTS.md, reference these files so Codex reads them when needed:

## Reference Documentation
- For SQL syntax rules, read docs/dremio-sql-reference.md
- For team table schemas, read docs/team-schemas.md
- For Python SDK patterns, read docs/dremioframe-patterns.md

Populate the knowledge files with your actual table schemas exported from Dremio, team-specific SQL patterns, and dremioframe code snippets for common operations.

Directory-Level Overrides

For monorepos, use AGENTS.override.md in each subdirectory to provide namespace-specific context. The parent AGENTS.md sets the Dremio conventions; the overrides specify which schemas and tables are relevant to each sub-project.

Using Dremio with Codex: Practical Use Cases

Once Dremio is connected, Codex becomes a data engineering assistant in your terminal. Here are detailed examples.

Ask Natural Language Questions About Your Data

Type a question directly in Codex and get answers from production data:

"What were our top 5 underperforming regions last quarter? Compare to the same quarter last year and suggest which metrics to investigate."

Codex discovers your tables via MCP, writes a multi-step SQL analysis, runs it against Dremio, and returns a structured answer. You get insights from production data without opening the Dremio UI.

Follow up with deeper investigation:

"For the worst-performing region, break down the decline by product category. Is it a demand issue or a fulfillment issue? Show return rates and delivery times alongside revenue."

Codex maintains session context and uses the AGENTS.md conventions to write correct Dremio SQL. The layered guidance system means your global Dremio rules apply automatically.

This pattern is especially powerful for engineers who live in the terminal. You can explore data, validate hypotheses, and generate insights without switching to a browser-based BI tool.

Build a Locally Running Dashboard

Ask Codex to create a complete visualization:

"Query Dremio's gold-layer financial views for revenue, expenses, and margins by department. Build a local HTML dashboard with D3.js charts showing trends, a summary table, and conditional formatting for over/under budget departments. Add a dark theme and filter controls."

Codex will:

Use MCP to discover gold-layer financial views and their schemas
Write and execute SQL queries for each metric
Generate an HTML file with D3.js interactive visualizations
Add conditional formatting (green/red) for budget variance
Include filter dropdowns for department and date range
Save the complete file to your project

Open it in a browser for an interactive financial dashboard. No server required. Re-run the prompt weekly with fresh data from Dremio.

Create a Data Exploration App

Build an interactive tool for your team:

"Create a Flask app with a REST API that proxies queries to Dremio through dremioframe. Add a React frontend with a table browser, column statistics view, and a SQL sandbox where I can run ad-hoc queries. Include authentication with API keys."

Codex scaffolds the full-stack app with:

Flask backend with dremioframe connection pooling
React frontend with schema browser and SQL editor
API key middleware for access control
docker-compose.yml for easy deployment
Proper project structure with requirements.txt and package.json

This pattern lets you create internal data tools quickly without a formal development cycle.

Generate Data Pipeline Code

Automate your ETL workflows:

"Write a Python pipeline using dremioframe that incrementally processes new customer records. Create bronze views for raw data with TIMESTAMP casts, silver views with deduplication and email validation, and gold views with customer segmentation logic using CASE WHEN expressions. Add logging, error handling, and a summary report at the end."

Codex follows the Dremio conventions from your AGENTS.md and produces production-ready pipeline code. The AGENTS.md cross-tool portability means the same conventions apply whether you run this from Codex, OpenCode, or OpenWork.

Build API Endpoints Over Dremio Data

Serve lakehouse data to other applications:

"Create a FastAPI service that connects to Dremio and serves customer analytics. Add endpoints for cohort analysis, retention metrics, and revenue forecasting. Include request validation, response caching, and health checks."

Codex generates a complete API server ready for uvicorn main:app --reload.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time access
AGENTS.md	10 minutes	Convention enforcement, doc references, portable config	Teams needing cross-tool consistency
Pre-Built Skills	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Quick start with broad coverage
Custom Config	30+ minutes	Tailored to your schemas, patterns, and monorepo layout	Mature teams with project-specific needs

These approaches stack. Start with the MCP server for live data access, add an AGENTS.md for Dremio conventions, and supplement with knowledge files as your team identifies recurring patterns. The layered guidance system in Codex (global, project, nested overrides) makes it easy to manage Dremio context at every level of your project hierarchy.

If your team uses multiple AI coding tools, invest in the AGENTS.md approach first. It gives you a single Dremio configuration that works across tools, and you can layer in MCP for live data access from whichever agent you are using at the time.

Get Started

Sign up for Dremio Cloud free for 30 days ($400 in compute credits included).
Find your project's MCP endpoint in Project Settings > Info.
Add it to Codex's MCP configuration.
Clone dremio-agent-skill and copy the AGENTS.md to your project root.
Start Codex and ask it to explore your Dremio catalog.

Dremio's Agentic Lakehouse delivers the three things Codex needs to write accurate analytics code: the semantic layer provides business context, query federation provides universal data access, and Reflections provide interactive speed. The MCP server bridges them, and AGENTS.md teaches the agent your team's conventions.

For more on the Dremio MCP Server, see the official documentation or take the free Dremio MCP Server course on Dremio University.

How to Use Dremio with JetBrains AI Assistant: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 16:00:00 GMT

JetBrains AI Assistant is built into IntelliJ IDEA, PyCharm, DataGrip, and every JetBrains IDE. It provides AI chat, inline code generation, multi-file refactoring, and agentic background workers that can autonomously execute multi-step tasks. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.

Connecting them gives the AI Assistant the context it needs to write accurate Dremio SQL, generate data pipelines, and build applications against your lakehouse. JetBrains IDEs are especially strong for data engineering: DataGrip provides native database tooling, IntelliJ supports full-stack development, and PyCharm is the standard for Python data work. Adding Dremio context to the AI Assistant turns these IDEs into data-aware development environments.

A unique feature of the JetBrains ecosystem is its dual MCP role: the AI Assistant acts as an MCP client (connecting to external servers like Dremio), and the IDE itself can also act as an MCP server (exposing IDE tools to other AI clients).

This post covers four approaches, ordered from quickest setup to most customizable.

Setting Up JetBrains AI Assistant

If you do not already have JetBrains AI Assistant:

Install a JetBrains IDE — IntelliJ IDEA, PyCharm, DataGrip, or any other JetBrains IDE from jetbrains.com. Community editions are free; Ultimate editions require a subscription.
Activate AI Assistant — AI Assistant is included with JetBrains IDE subscriptions (2025.1+). Go to Settings > Plugins and ensure "AI Assistant" is enabled.
Sign in with your JetBrains account to activate the AI quota.
Open the AI Chat by clicking the AI Assistant icon in the right sidebar or pressing Alt+Enter on a code selection.

JetBrains AI Assistant supports multiple LLM providers. You can use JetBrains-hosted models, connect your own API keys for Anthropic or OpenAI, or run local models via OpenAI-compatible servers for privacy-sensitive environments.

Approach 1: Connect the Dremio Cloud MCP Server

Every Dremio Cloud project ships with a built-in MCP server. JetBrains AI Assistant supports MCP as a client starting with version 2025.1.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup. For JetBrains, you configure the MCP connection through the IDE settings.

Find Your Project's MCP Endpoint

Log into Dremio Cloud and navigate to Project Settings > Info. Copy the MCP server URL.

Set Up OAuth in Dremio Cloud

Go to Settings > Organization Settings > OAuth Applications.
Click Add Application and enter a name (e.g., "JetBrains MCP").
Add the appropriate redirect URIs.
Save and copy the Client ID.

Configure JetBrains MCP Connection

Go to Settings > Tools > AI Assistant > Model Context Protocol (MCP). Click Add and select the transport type:

Streamable HTTP: For Dremio Cloud's hosted MCP server. Enter the MCP URL directly.
STDIO: For the self-hosted dremio-mcp server. Enter the command and arguments.

For HTTP configuration:

Name: Dremio
Type: Streamable HTTP
URL: https://YOUR_PROJECT_MCP_URL

After adding the server, the AI Assistant has access to Dremio's MCP tools:

GetUsefulSystemTableNames returns available tables.
GetSchemaOfTable returns column definitions.
GetDescriptionOfTableOrSchema pulls catalog descriptions.
GetTableOrViewLineage shows data lineage.
RunSqlQuery executes SQL and returns results.

Test by asking the AI chat: "What tables are available in Dremio?"

Self-Hosted Alternative

For Dremio Software deployments, configure the dremio-mcp server as STDIO transport:

Name: Dremio
Type: STDIO
Command: uv
Arguments: run --directory /path/to/dremio-mcp dremio-mcp-server run

Approach 2: Use Project Rules for Dremio Context

JetBrains AI Assistant supports project-specific rules through markdown files in .aiassistant/rules/. These files provide persistent AI instructions scoped to your project.

Create Project Rules

Create .aiassistant/rules/dremio.md:

# Dremio SQL Conventions

This project uses Dremio Cloud as its lakehouse platform.

## SQL Rules
- Use CREATE FOLDER IF NOT EXISTS (not CREATE NAMESPACE or CREATE SCHEMA)
- Tables in the Open Catalog use folder.subfolder.table_name
- External federated sources use source_name.schema.table_name
- Cast DATE to TIMESTAMP for consistent joins
- Use TIMESTAMPDIFF for duration calculations

## Credentials
- Never hardcode Personal Access Tokens. Use environment variable: DREMIO_PAT
- Cloud endpoint: environment variable DREMIO_URI

## Terminology
- Call it "Agentic Lakehouse", not "data warehouse"
- "Reflections" are pre-computed optimizations, not "materialized views"

You can also set rules via the IDE: Settings > Tools > AI Assistant > Project Rules.

Custom Prompts

Create reusable prompts in the Prompt Library (Settings > Tools > AI Assistant > Prompt Library). For example, create a "Dremio SQL Review" prompt that validates SQL against Dremio conventions before execution. These prompts are available from the AI Actions menu and can be invoked on selected code.

DataGrip Integration

If you use DataGrip or the Database plugin in IntelliJ, you can connect directly to Dremio as a JDBC data source. The AI Assistant then has access to your live schema through the IDE's built-in database tools, complementing the MCP-based approach.

Approach 3: Install Pre-Built Dremio Skills and Docs

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill (Community)

The dremio-agent-skill repository provides knowledge files and rules:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cd dremio-agent-skill
./install.sh

Copy the knowledge files into your project's .aiassistant/rules/ directory and reference them from your project rules.

dremio-agent-md (Community)

The dremio-agent-md repository provides a protocol file and sitemaps:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Reference it in your project rules:

For Dremio SQL validation, read DREMIO_AGENT.md in ./dremio-agent-md/.

Approach 4: Build Your Own Project Rules

Create a comprehensive rules setup in .aiassistant/rules/:

.aiassistant/rules/
  dremio-sql.md           # SQL conventions
  dremio-python.md        # dremioframe patterns
  dremio-schemas.md       # Team table schemas
  dremio-api.md           # REST API patterns

Export your actual schemas from Dremio and keep them as a rule file. The AI Assistant reads all files in the rules/ directory and applies them to relevant interactions.

Using Dremio with JetBrains AI: Practical Use Cases

Once Dremio is connected, the AI Assistant becomes a data-aware coding partner across all JetBrains IDEs.

Ask Natural Language Questions About Your Data

In the AI Chat panel, ask questions about your lakehouse:

"What were our top 10 accounts by contract value last quarter? Break down by industry vertical and show renewal rates."

The AI uses MCP to discover tables, writes the SQL, and returns results. In DataGrip, you can execute the generated SQL directly in the query console for additional exploration.

Follow up:

"For accounts with renewal rates below 70%, pull their support ticket history and calculate average resolution time. Cross-reference with product usage metrics."

The AI maintains conversation context and generates multi-table joins.

Build a Locally Running Dashboard

Ask the AI to generate a dashboard project:

"Query Dremio gold-layer views for revenue, customer metrics, and churn data. Create an HTML dashboard with ECharts. Include date filters, dark theme, and regional drill-down. Generate separate files."

The AI generates the complete dashboard. In IntelliJ or WebStorm, you can preview the HTML directly in the IDE's built-in browser.

Create a Data Exploration App

Generate a data tool:

"Create a Streamlit app connected to Dremio via dremioframe. Include schema browsing, SQL query editor with syntax highlighting, data preview with pagination, and CSV download. Generate requirements.txt."

In PyCharm, the AI generates the app and you can run it directly from the IDE with integrated debugging.

Generate Data Pipeline Scripts

Use the AI for data engineering:

"Write a Medallion Architecture pipeline using dremioframe. Bronze: ingest raw data. Silver: deduplicate, validate, standardize timestamps. Gold: business metrics and KPIs. Include logging and error handling."

The AI generates the pipeline code following your project rules. PyCharm's debugger lets you step through the pipeline against live Dremio data.

Build API Endpoints Over Dremio Data

Scaffold backend services:

"Build a FastAPI app that serves Dremio analytics through REST endpoints. Add customer segments, revenue by region, and product trends. Include Pydantic models and OpenAPI docs."

IntelliJ's HTTP client lets you test the endpoints directly from the IDE.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time access
Project Rules	10 minutes	Convention enforcement, persistent AI context	Teams with specific standards per IDE
Pre-Built Skills	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Quick start with broad coverage
Custom Rules	30+ minutes	Tailored schemas, custom prompts, team conventions	Mature teams with DataGrip/PyCharm workflows

Start with the MCP server for live data access. Add project rules for conventions. Use DataGrip's native Dremio connection for schema exploration alongside MCP.

Get Started

Sign up for a free Dremio Cloud trial (30 days, $400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Add it in Settings > Tools > AI Assistant > MCP.
Create .aiassistant/rules/dremio.md with your SQL conventions.
Open AI Chat and ask it to explore your Dremio catalog.

Dremio's Agentic Lakehouse gives the JetBrains AI Assistant accurate data context, and the IDE's native database tooling provides complementary schema exploration and SQL execution.

For more on the Dremio MCP Server, check out the official documentation or enroll in the free Dremio MCP Server course on Dremio University.

How to Use Dremio with Google Antigravity: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 15:00:00 GMT

Google Antigravity is an agent-first IDE built by Google DeepMind. Its autonomous agents plan multi-step tasks, write code, browse documentation, and iterate without constant hand-holding. Dremio is a unified lakehouse platform that provides the business context, universal data access, and interactive query speed that AI agents need to produce accurate analytics.

Connecting the two gives your Antigravity agents something most coding agents lack: direct access to your data catalog, table schemas, business logic encoded in views, and the correct SQL dialect for Dremio's query engine. Without it, the agent guesses at table names and hallucinates SQL functions. With it, the agent writes queries that actually run.

Antigravity's skill system is a particularly strong fit for Dremio integration. Skills load on demand based on semantic matching, so Dremio knowledge enters the context only when the agent needs it. This keeps the context window efficient for tasks that have nothing to do with data, while still providing deep Dremio expertise when you shift to analytics work.

This post walks through four integration approaches. Each one adds a different kind of context, and they combine well. You can start with the simplest option and layer in more approaches as your team's Dremio usage grows.

Setting Up Google Antigravity

If you do not already have Antigravity installed:

Download Antigravity from the Google DeepMind tools page or your organization's approved software catalog.
Install it by following the platform-specific instructions (available for macOS, Linux, and Windows).
Open a project by launching Antigravity and pointing it to your project directory.
Configure your AI model by adding your API key or connecting your Google Cloud account in the IDE settings.

Antigravity's agent-first design means it can plan multi-step tasks, execute shell commands, browse documentation, and iterate autonomously. Its skill system and rules engine give you fine-grained control over how agents behave.

Approach 1: Connect the Dremio Cloud MCP Server

The Model Context Protocol (MCP) is an open standard for AI tools to call external services. Dremio Cloud includes a built-in MCP server in every project, and Antigravity supports MCP natively through its IDE settings.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup. Since Antigravity uses its own MCP configuration, you will configure the connection through the IDE settings instead.

Find Your Project's MCP Endpoint

Log into Dremio Cloud and open your project. Go to Project Settings > Info. The MCP server URL is displayed on the project overview page. Copy it.

Set Up OAuth in Dremio Cloud

The hosted MCP server uses OAuth to authenticate connections. Your existing Dremio access controls apply to every query your Antigravity agent runs.

Navigate to Settings > Organization Settings > OAuth Applications.
Click Add Application and enter a name like "Antigravity MCP".
Add the appropriate redirect URI for your Antigravity setup.
Save and copy the Client ID.

Configure Antigravity's MCP Connection

In Antigravity, open the MCP settings panel from the IDE preferences. Add a new MCP server with your Dremio project URL and the OAuth client credentials. The agent will now have access to Dremio's MCP tools:

GetUsefulSystemTableNames returns available tables with descriptions.
GetSchemaOfTable returns column names, types, and metadata.
GetDescriptionOfTableOrSchema pulls wiki descriptions and labels from the catalog.
GetTableOrViewLineage shows upstream data dependencies.
RunSqlQuery executes SQL and returns results as JSON.

Test the connection by asking your Antigravity agent: "What tables are available in Dremio?" The agent will call GetUsefulSystemTableNames and return your catalog contents.

Self-Hosted Alternative

For Dremio Software deployments, use the open-source dremio-mcp server. Clone the repo, configure it with your Dremio instance URL and a Personal Access Token (PAT), then point Antigravity's MCP settings to the local server:

git clone https://github.com/dremio/dremio-mcp
cd dremio-mcp
uv run dremio-mcp-server config create dremioai \
  --uri https://your-dremio-instance.com \
  --pat YOUR_PERSONAL_ACCESS_TOKEN

In Antigravity's MCP settings, configure the server to run via the local command:

{
  "command": "uv",
  "args": [
    "run", "--directory", "/path/to/dremio-mcp",
    "dremio-mcp-server", "run"
  ]
}

The self-hosted server supports three modes: FOR_DATA_PATTERNS for data exploration and SQL generation (default), FOR_SELF for system performance analysis, and FOR_PROMETHEUS for correlating Dremio metrics with your monitoring stack.

Approach 2: Use SKILL.md and Agent Rules for Dremio Context

Antigravity's defining feature is its skill system. Skills are reusable knowledge packages that agents discover and load on demand. A skill is a directory containing a SKILL.md file with YAML frontmatter for discovery and markdown instructions for the agent.

The key difference from context files in other tools: Antigravity skills are loaded only when relevant. The agent reads the skill's name and description from the YAML frontmatter, semantically matches them against your prompt, and activates the skill only when it is needed. This avoids wasting context tokens on instructions the agent does not need for the current task.

This architecture is called progressive disclosure. A tool like Claude Code loads CLAUDE.md into every session whether you need it or not. Antigravity loads skills selectively. For teams that use Dremio for some projects and not others, this means zero overhead on non-Dremio work.

How SKILL.md Works

A SKILL.md file has two parts:

---
name: Dremio Conventions
description: SQL syntax, REST API patterns, and credential handling for Dremio Cloud
---

# Dremio Conventions

## SQL Rules
- Use CREATE FOLDER IF NOT EXISTS (not CREATE NAMESPACE)
- Tables in the Open Catalog use folder.subfolder.table_name without a catalog prefix
- External sources use source_name.schema.table_name
- Cast DATE to TIMESTAMP for consistent joins

## Credentials
- Never hardcode PATs. Use environment variable DREMIO_PAT
- Dremio Cloud endpoint: environment variable DREMIO_URI

## Reference
- For SQL syntax validation, read knowledge/sql-reference.md
- For REST API endpoints, read knowledge/rest-api.md

Place this in .agent/skills/dremio/SKILL.md for workspace scope or ~/.agent/skills/dremio/SKILL.md for global scope.

Agent Rules for Always-On Guidance

Skills activate on demand. For instructions that should apply to every session regardless of the prompt, use Antigravity's rules system. Place markdown files in .agent/rules/:

# .agent/rules/dremio-sql.md

When writing Dremio SQL:
- Never use CREATE SCHEMA or CREATE NAMESPACE. Dremio uses CREATE FOLDER IF NOT EXISTS.
- Always validate function names against the Dremio SQL reference before including them.
- Use TIMESTAMPDIFF for duration calculations, not DATEDIFF.
- Dremio is not a data warehouse. It is an Agentic Lakehouse platform.

Rules load at session start, similar to CLAUDE.md in Claude Code. Use rules for hard constraints (like SQL dialect rules) and skills for reference knowledge (like API documentation).

Workflows for Repetitive Dremio Tasks

Antigravity also supports workflows in .agent/workflows/. These are saved prompts the agent follows step by step. For example:

# .agent/workflows/dremio-data-model.md
---
description: Create a bronze-silver-gold data model in Dremio
---

1. Read the Dremio skill for SQL conventions
2. Create folders for bronze, silver, and gold layers
3. Create bronze views with column renames and TIMESTAMP casts
4. Create silver views joining bronze views with business logic
5. Create gold views with CASE WHEN classifications
6. Enable AI-generated wikis on gold views

Approach 3: Install Pre-Built Dremio Skills and Docs

Two community-supported open-source repositories provide ready-made Dremio context. Antigravity has first-class support for the skill-based approach.

Official vs. Community Resources: Dremio offers an official Claude Code plugin for Claude-based tools, and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill: Native Antigravity Skill (Community)

The dremio-agent-skill repository is designed for tools like Antigravity. It contains a complete dremio-skill/ directory with SKILL.md, comprehensive knowledge/ files (CLI, Python SDK, SQL, REST API), and configuration files for other tools.

Install it globally:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cd dremio-agent-skill
./install.sh

Choose Global Install (Symlink) when prompted. This creates a symlink from the repo's dremio-skill/ directory to ~/.agent/skills/, making the skill available in every Antigravity workspace. When you pull updates to the repo, the skill updates automatically.

After installation, start a new Antigravity session and ask it to scan for available skills. The agent will discover the Dremio skill by its name and description, and load it whenever you ask Dremio-related questions.

For team projects, choose Local Project Install (Copy) instead. This copies the skill into your project and sets up .agent symlinks so every team member who clones the repo gets the same context.

dremio-agent-md: Documentation Protocol (Community)

The dremio-agent-md repository provides a DREMIO_AGENT.md protocol file and browsable sitemaps of the Dremio documentation.

Clone it and tell your Antigravity agent to read it:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Then instruct the agent: "Read DREMIO_AGENT.md in the dremio-agent-md directory. Use the sitemaps in dremio_sitemaps/ to verify Dremio syntax before generating any SQL."

This approach is useful when you need the agent to cross-reference specific documentation pages rather than rely on pre-packaged knowledge files.

Approach 4: Build Your Own Dremio Skill

If the pre-built skill does not fit your workflow, build a custom one. Antigravity's skill system makes this straightforward.

Create the Skill Structure

.agent/skills/my-dremio/
  SKILL.md
  knowledge/
    sql-conventions.md
    team-schemas.md
    dremioframe-patterns.md

Write the SKILL.md

---
name: Team Dremio Skill
description: SQL conventions, table schemas, and dremioframe patterns for our analytics lakehouse
---

# Team Dremio Skill

## SQL Standards
- All tables are under the analytics namespace
- Bronze: analytics.bronze.*, Silver: analytics.silver.*, Gold: analytics.gold.*
- Always use TIMESTAMP, never DATE
- Validate function names against knowledge/sql-conventions.md

## Authentication
- Use env var DREMIO_PAT for tokens
- Cloud endpoint: env var DREMIO_URI

## Common Tasks
- For bulk data operations, use dremioframe patterns in knowledge/dremioframe-patterns.md
- For table schemas, check knowledge/team-schemas.md

Populate Knowledge Files

Export your actual table schemas from Dremio and save them as markdown in the knowledge/ directory. Include dremioframe code snippets your team uses frequently, REST API call patterns for your CI/CD pipeline, and SQL examples that follow your naming conventions.

The advantage of a custom skill over a generic rules file: skills activate based on semantic matching. When you ask about a completely unrelated topic, the Dremio skill stays out of the context window. When you ask about data pipelines or SQL, the agent pulls it in automatically.

Using Dremio with Antigravity: Practical Use Cases

Once Dremio is connected, Antigravity's agents can execute complete data projects autonomously. Here are detailed examples.

Ask Natural Language Questions About Your Data

Ask your Antigravity agent questions about your lakehouse in plain English:

"What is the average order value by product category for the last 6 months? Show me which categories are trending up."

The agent uses MCP to discover relevant tables, writes and runs the SQL against Dremio, and returns formatted results with analysis. No SQL required.

Take it further with multi-step analysis:

"For the categories trending up, pull the top 5 products in each and compare their margins. Are we making more revenue but at lower margins?"

Antigravity's skill system loads the Dremio conventions automatically when it detects a data-related question, so the SQL it generates follows your team's standards without you needing to remind it.

Build a Locally Running Dashboard

Give the agent a broader task:

"Query our gold-layer customer analytics views in Dremio. Build a local HTML dashboard with Plotly.js charts showing customer lifetime value distribution, churn rates by cohort, and retention curves. Include date range filters and a dark theme."

Antigravity will:

Activate the Dremio skill to understand your SQL conventions
Use MCP to discover gold-layer views and their schemas
Write and execute the SQL queries
Generate an HTML file with Plotly.js interactive charts
Add filter controls and a responsive layout
Save it to your workspace

Open the HTML file in a browser for a complete dashboard running from a local file. The Plotly.js charts support zoom, pan, hover tooltips, and export to PNG.

Create a Data Exploration App

Ask for an interactive tool:

"Build a Streamlit app that connects to Dremio using dremioframe. Add a sidebar for browsing schemas and tables, a detail view showing table schemas and wiki descriptions, a SQL query editor with syntax highlighting, and a results panel with pagination and CSV download."

Antigravity writes the full Python application with:

Dremio catalog browser using the MCP connection for live schema data
SQL editor with autocomplete based on discovered table names
Paginated results display with export options
Connection management using environment variables

Run streamlit run app.py and your team has a local data explorer without waiting for a BI tool deployment.

Automate Data Workflows with Antigravity Workflows

Use Antigravity's workflow system to create repeatable Dremio operations:

"Using the Dremio skill, write a Python script that creates a bronze-silver-gold view hierarchy for our new user events table. Follow the Medallion Architecture patterns. Bronze should rename columns to snake_case and cast dates. Silver should deduplicate and validate required fields. Gold should aggregate daily active users and session duration by segment."

The agent references the Dremio skill for conventions and produces structured SQL and Python code. Save the prompt as an Antigravity workflow in .agent/workflows/new-data-model.md so any team member can run it for new tables.

Build API Endpoints Over Dremio Data

Create backend services:

"Build a Flask API that queries Dremio's gold-layer views. Create endpoints for customer segments, revenue trends, and product performance. Include caching with a 5-minute TTL and rate limiting. Generate OpenAPI docs."

Antigravity generates the full application with proper error handling, connection pooling via dremioframe, and production-ready configuration.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time access
SKILL.md + Rules	15 minutes	Convention enforcement, on-demand doc references	Teams with specific SQL standards
Pre-Built Skill	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Quick start with broad coverage
Custom Skill	30+ minutes	Tailored to your schemas, patterns, workflows	Mature teams with specific needs

Combine them for the strongest setup. Use the MCP server for live data, a pre-built skill for general Dremio knowledge, rules for hard SQL constraints, and a custom skill for your team's specific schemas and patterns.

If you are evaluating Dremio for the first time, start with the MCP server. It takes five minutes and gives you immediate querying capabilities. As you develop team conventions, add rules files for the constraints that should apply universally. Once you have a stable set of patterns, package them into a custom skill that your entire team can install.

Get Started

Sign up for Dremio Cloud free for 30 days ($400 in compute credits included).
Find your project's MCP endpoint under Project Settings > Info.
Add it in Antigravity's MCP settings panel.
Clone dremio-agent-skill and run ./install.sh with global symlink mode.
Start a new Antigravity session and ask it to explore your Dremio catalog.

Dremio's Agentic Lakehouse provides the three things Antigravity agents need for accurate analytics: the semantic layer delivers business context, query federation delivers universal data access, and Reflections deliver interactive speed. The MCP server connects them, and skills teach the agent your team's conventions.

For more on the Dremio MCP Server, see the official documentation or take the free Dremio MCP Server course on Dremio University.

How to Use Dremio with GitHub Copilot: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 14:00:00 GMT

GitHub Copilot is the most widely adopted AI coding assistant, integrated into VS Code, JetBrains IDEs, and the GitHub platform. Its agent mode allows Copilot to plan and execute multi-step coding tasks, run terminal commands, and interact with external tools through MCP. The Copilot CLI extends agentic development to the terminal. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.

Connecting them gives Copilot's agent mode the context it needs to write accurate Dremio SQL, generate data pipelines, and build applications against your lakehouse. This is significant because of Copilot's massive user base: if you already use Copilot for code completion and chat, adding Dremio context turns it into a data-aware development partner without switching tools.

Copilot's copilot-instructions.md file and .vscode/mcp.json configuration make it straightforward to integrate project-specific Dremio conventions and live data access into your workflow.

This post covers four approaches, ordered from quickest setup to most customizable.

Setting Up GitHub Copilot

If you do not already have GitHub Copilot:

Sign up for GitHub Copilot at github.com/features/copilot. Individual ($10/month), Business ($19/user/month), and Enterprise ($39/user/month) plans are available. Free tier includes limited completions.
Install VS Code from code.visualstudio.com if not already installed.
Install the GitHub Copilot extension from the VS Code marketplace (search "GitHub Copilot").
Sign in with your GitHub account when prompted.
Enable agent mode by clicking the Copilot chat icon and selecting "Agent" from the mode dropdown (available in VS Code 1.99+).

For terminal usage, install the Copilot CLI (gh copilot) through the GitHub CLI:

gh extension install github/gh-copilot

Approach 1: Connect the Dremio Cloud MCP Server

Every Dremio Cloud project ships with a built-in MCP server. Copilot agent mode supports MCP natively through workspace configuration files.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup. For Copilot, you configure the MCP connection through .vscode/mcp.json.

Find Your Project's MCP Endpoint

Log into Dremio Cloud and navigate to Project Settings > Info. Copy the MCP server URL.

Set Up OAuth in Dremio Cloud

Go to Settings > Organization Settings > OAuth Applications.
Click Add Application and enter a name (e.g., "Copilot MCP").
Add the appropriate redirect URIs.
Save and copy the Client ID.

Configure Copilot's MCP Connection

Create .vscode/mcp.json in your project root:

{
  "servers": {
    "dremio": {
      "type": "http",
      "url": "https://YOUR_PROJECT_MCP_URL"
    }
  }
}

You can also configure MCP servers in your VS Code user settings:

{
  "mcp": {
    "servers": {
      "dremio": {
        "type": "http",
        "url": "https://YOUR_PROJECT_MCP_URL"
      }
    }
  }
}

Reload VS Code. Copilot agent mode now has access to Dremio's MCP tools:

GetUsefulSystemTableNames returns available tables with descriptions.
GetSchemaOfTable returns column names, types, and metadata.
GetDescriptionOfTableOrSchema pulls wiki descriptions from the catalog.
GetTableOrViewLineage shows upstream dependencies.
RunSqlQuery executes SQL and returns results as JSON.

Test the connection by opening Copilot Chat in agent mode and asking: "What tables are available in Dremio?" The agent will call GetUsefulSystemTableNames and return your catalog contents.

Self-Hosted Alternative

For Dremio Software deployments, use the open-source dremio-mcp server:

{
  "servers": {
    "dremio": {
      "type": "stdio",
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

Enterprise Policy Controls

For organizations, GitHub administrators can manage MCP server access through organization policies. This lets teams standardize on approved Dremio MCP connections while preventing unauthorized data access.

Approach 2: Use copilot-instructions.md for Dremio Context

Copilot reads custom instructions from .github/copilot-instructions.md in your repository. This file is loaded into every Copilot interaction, providing persistent project context.

Repository-Level Instructions

Create .github/copilot-instructions.md:

# Dremio SQL Conventions

This project uses Dremio Cloud as its lakehouse platform.

## SQL Rules
- Use CREATE FOLDER IF NOT EXISTS (not CREATE NAMESPACE or CREATE SCHEMA)
- Tables in the Open Catalog use folder.subfolder.table_name
- External federated sources use source_name.schema.table_name
- Cast DATE to TIMESTAMP for consistent joins
- Use TIMESTAMPDIFF for duration calculations

## Credentials
- Never hardcode Personal Access Tokens. Use environment variable: DREMIO_PAT
- Cloud endpoint: environment variable DREMIO_URI

## Terminology
- Call it "Agentic Lakehouse", not "data warehouse"
- "Reflections" are pre-computed optimizations, not "materialized views"

Pattern-Specific Instructions

Copilot also supports .instructions files with YAML glob patterns for targeted application:

Create .github/instructions/dremio-sql.instructions.md:

---
applyTo: "**/*.sql"
---

When writing SQL for Dremio:
- Validate function names against the Dremio SQL reference
- Use TIMESTAMPDIFF for duration calculations
- Cast DATE columns to TIMESTAMP before joins

Create .github/instructions/dremio-python.instructions.md:

---
applyTo: "**/*.py"
---

When writing Python code that uses dremioframe:
- Import as: from dremioframe import DremioConnection
- Use environment variables for credentials
- Always close connections in a finally block

This scoping is similar to Cursor's .cursor/rules/*.mdc pattern matching.

Approach 3: Install Pre-Built Dremio Skills and Docs

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill (Community)

The dremio-agent-skill repository provides knowledge files and a .cursorrules file:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cd dremio-agent-skill
./install.sh

Reference the knowledge files from your copilot-instructions.md:

For Dremio SQL conventions, read the knowledge files in dremio-skill/knowledge/.

dremio-agent-md (Community)

The dremio-agent-md repository provides a protocol file and documentation sitemaps:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Reference it in .github/copilot-instructions.md:

For Dremio documentation, read DREMIO_AGENT.md in ./dremio-agent-md/.

Approach 4: Build Your Own copilot-instructions.md

Create a comprehensive instruction file with your team's Dremio environment:

# Team Dremio Context

## Environment
- Lakehouse: Dremio Cloud (analytics project)
- Catalog: Apache Polaris-based Open Catalog
- Architecture: Medallion (bronze → silver → gold)

## Table Schemas
For exact column definitions, read ./docs/table-schemas.md

## SQL Standards
- Bronze: raw.*, Silver: cleaned.*, Gold: analytics.*
- Always use TIMESTAMP, never DATE
- Validate function names against ./docs/dremio-sql-reference.md

## Python SDK
- Use dremioframe for all Dremio connections
- Patterns: read ./docs/dremioframe-patterns.md

Using Dremio with GitHub Copilot: Practical Use Cases

Once Dremio is connected, Copilot agent mode can execute complete data projects. Here are detailed examples.

Ask Natural Language Questions About Your Data

In agent mode, ask Copilot questions about your data:

"What were our top 10 customers by lifetime value? Show their order frequency and most recent purchase date."

Copilot agent mode uses MCP to discover tables, writes the SQL, runs it, and returns formatted results. Because it operates within VS Code, you can immediately use the results in your code.

Follow up with analysis:

"For customers with declining order frequency, correlate with support ticket volume. Are our high-value customers churning?"

Copilot maintains context across the conversation and generates cross-table queries automatically.

Build a Locally Running Dashboard

Ask Copilot in agent mode:

"Query our Dremio gold-layer views for revenue metrics, then create an HTML dashboard with Chart.js. Include monthly trends, regional breakdown, and top product charts. Add date filters and a dark theme. Save as separate HTML, CSS, and JS files."

Copilot agent mode will:

Call MCP to discover views and schemas
Execute queries and collect results
Generate index.html, styles.css, and app.js
Wire everything together with Chart.js

Open index.html in a browser for a working dashboard. Since this all happens in VS Code, you can iterate on the design with inline edits.

Create a Data Exploration App

Build interactive tools:

"Create a Streamlit app connected to Dremio via dremioframe. Include schema browsing, data preview with pagination, SQL query editor, and CSV export. Generate all files and a README."

Copilot generates the full application. Run streamlit run app.py for a local data explorer.

Generate Data Pipeline Scripts

Use inline completions for data engineering:

Write a comment: # Medallion pipeline for product_events: bronze ingestion, silver cleaning, gold aggregation

Copilot generates the complete pipeline following your copilot-instructions.md conventions. Agent mode can also run the generated code against your Dremio instance to validate it.

Build API Endpoints Over Dremio Data

Create backend services:

"Build a FastAPI app that serves Dremio gold-layer data through REST endpoints. Add customer analytics, revenue by region, and product performance. Include Pydantic models, caching, and OpenAPI docs."

Copilot generates the complete API. Run uvicorn main:app --reload for a local server.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time access
copilot-instructions.md	10 minutes	Convention enforcement, pattern-specific rules	Teams with repository-wide standards
Pre-Built Skills	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Quick start with broad coverage
Custom Instructions	30+ minutes	Tailored schemas, patterns, and team conventions	Mature teams with project-specific needs

Start with the MCP server for immediate value. Add copilot-instructions.md for conventions. Use .instructions files for pattern-specific rules.

Get Started

Sign up for a free Dremio Cloud trial (30 days, $400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Create .vscode/mcp.json with your Dremio MCP server.
Create .github/copilot-instructions.md with Dremio conventions.
Open Copilot in agent mode and ask it to explore your Dremio catalog.

Dremio's Agentic Lakehouse gives Copilot accurate data context. Combined with Copilot's massive user base and VS Code integration, this is the lowest-friction path to AI-powered data development for most teams.

For more on the Dremio MCP Server, check out the official documentation or enroll in the free Dremio MCP Server course on Dremio University.

How to Use Dremio with Gemini CLI: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 13:00:00 GMT

Gemini CLI is Google's open-source terminal-based AI agent. It runs directly in your terminal, powered by Gemini models with a 1-million token context window. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.

Connecting them gives Gemini CLI the data context it needs to write accurate Dremio SQL, generate pipeline scripts, and build applications against your lakehouse. The 1-million token context window is a significant advantage: Gemini CLI can hold your entire project, documentation, and Dremio schema context simultaneously without the context limitations that constrain other agents.

Gemini CLI's GEMINI.md context file system is similar to CLAUDE.md in Claude Code. It loads project-specific instructions at session start and supports hierarchical scoping from global defaults to project-specific overrides. The tool also supports MCP natively, Google Search grounding for real-time documentation lookups, and built-in file and shell tools.

This post covers four approaches, ordered from quickest setup to most customizable.

Setting Up Gemini CLI

If you do not already have Gemini CLI installed:

Install Node.js (version 18 or later) from nodejs.org.
Install Gemini CLI globally via npm:
```
npm install -g @anthropic-ai/gemini-cli
```
Or install from source via the GitHub repository.
Authenticate by running gemini in your terminal. On first launch, it will prompt you to sign in with your Google account. Gemini CLI is free to use with a Google account (rate-limited) or with a Gemini API key for higher throughput.
Verify the installation by asking a question: gemini "What is Apache Iceberg?"

Gemini CLI runs in your terminal and reads your project files for context. It can execute shell commands, edit files, browse the web via Google Search grounding, and interact with MCP servers.

Approach 1: Connect the Dremio Cloud MCP Server

The Model Context Protocol (MCP) is an open standard that lets AI tools call external services. Every Dremio Cloud project ships with a built-in MCP server. Gemini CLI supports MCP natively through its settings.json configuration.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup. For Gemini CLI, you configure the MCP connection through settings.json.

Find Your Project's MCP Endpoint

Log into Dremio Cloud and open your project. Navigate to Project Settings > Info. The MCP server URL is listed on the project overview page. Copy it.

Set Up OAuth in Dremio Cloud

Go to Settings > Organization Settings > OAuth Applications.
Click Add Application and enter a name (e.g., "Gemini CLI MCP").
Add the appropriate redirect URI for your setup.
Save and copy the Client ID.

Configure Gemini CLI's MCP Connection

Gemini CLI reads MCP server definitions from settings.json. You can configure this at two levels:

User-level: ~/.gemini/settings.json (applies to all projects)
Project-level: .gemini/settings.json (applies to the current project only)

Create or edit the settings file:

{
  "mcpServers": {
    "dremio": {
      "httpUrl": "https://YOUR_PROJECT_MCP_URL"
    }
  }
}

You can also add MCP servers using the CLI command:

gemini mcp add dremio --httpUrl "https://YOUR_PROJECT_MCP_URL"

Restart Gemini CLI. The agent now has access to Dremio's MCP tools:

GetUsefulSystemTableNames returns available tables with descriptions.
GetSchemaOfTable returns column names, types, and metadata.
GetDescriptionOfTableOrSchema pulls wiki descriptions from the catalog.
GetTableOrViewLineage shows upstream dependencies.
RunSqlQuery executes SQL and returns results as JSON.

Test the connection by asking: "What tables are available in Dremio?" Gemini CLI will call GetUsefulSystemTableNames and return your catalog contents.

Self-Hosted Alternative

For Dremio Software deployments, use the open-source dremio-mcp server:

git clone https://github.com/dremio/dremio-mcp
cd dremio-mcp
uv run dremio-mcp-server config create dremioai \
  --uri https://your-dremio-instance.com \
  --pat YOUR_PERSONAL_ACCESS_TOKEN

In your settings.json:

{
  "mcpServers": {
    "dremio": {
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

The self-hosted server supports three modes: FOR_DATA_PATTERNS for data exploration (default), FOR_SELF for system analysis, and FOR_PROMETHEUS for correlating metrics with monitoring.

Approach 2: Use GEMINI.md for Dremio Context

Gemini CLI auto-loads GEMINI.md from your project root at the start of every session. It works similarly to CLAUDE.md in Claude Code, providing persistent instructions that survive across conversations.

Hierarchical Context Loading

GEMINI.md supports hierarchical scoping:

Global: ~/.gemini/GEMINI.md applies to every project.
Project: GEMINI.md in the project root applies to that specific repo.
Subdirectory: GEMINI.md files in subdirectories provide additional context when working in those folders.

Project-level files override global ones. Subdirectory files add to the project context rather than replacing it.

Writing a Dremio-Focused GEMINI.md

# Project Context

This project uses Dremio Cloud as its lakehouse platform.

## Dremio SQL Conventions
- Use `CREATE FOLDER IF NOT EXISTS` (not CREATE NAMESPACE or CREATE SCHEMA)
- Tables in the Open Catalog use `folder.subfolder.table_name` without a catalog prefix
- External federated sources use `source_name.schema.table_name`
- Cast DATE columns to TIMESTAMP for consistent joins
- Use TIMESTAMPDIFF for duration calculations

## Credentials
- Never hardcode Personal Access Tokens. Use environment variable: DREMIO_PAT
- Dremio Cloud endpoint is in environment variable: DREMIO_URI

## API Reference
- REST API docs: https://docs.dremio.com/current/reference/api/
- SQL reference: https://docs.dremio.com/current/reference/sql/
- For detailed SQL validation, read ./dremio-docs/sql-reference.md

## Terminology
- Call it "Agentic Lakehouse", not "data warehouse"
- "Reflections" are pre-computed optimizations, not "materialized views"
- "Open Catalog" is built on Apache Polaris

Using Protocol Blocks for Gated Instructions

Gemini CLI supports <PROTOCOL> blocks within GEMINI.md for instructions that should only activate when specific conditions are met. This prevents context bloat:

<PROTOCOL>
When the user asks about Dremio SQL or data pipelines:
1. Read ./dremio-docs/sql-reference.md for syntax validation
2. Use Dremio SQL conventions defined above
3. Always verify function names exist in the reference before using them
</PROTOCOL>

Protocol blocks are a form of delayed instructions. Gemini CLI reads the protocol definition but only executes the instructions when the triggering condition is met. This is more efficient than loading all reference files at session start.

Google Search Grounding

Gemini CLI has built-in Google Search grounding, meaning it can look up real-time Dremio documentation during a session. You can instruct it in GEMINI.md:

## Documentation Strategy
- Before writing any Dremio SQL, use Google Search to verify the syntax
  against the latest Dremio documentation at docs.dremio.com
- If a function name is uncertain, search for it before including it

This is a unique advantage over other agents. Instead of relying solely on pre-loaded context or training data, Gemini CLI can verify syntax against live documentation.

Approach 3: Install Pre-Built Dremio Skills and Docs

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill (Community)

The dremio-agent-skill repository contains a complete skill directory with SKILL.md, knowledge files, and configuration files for multiple tools.

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cd dremio-agent-skill
./install.sh

For Gemini CLI, tell the agent to read the skill at session start:

"Read dremio-skill/SKILL.md and use the knowledge files in dremio-skill/knowledge/ for Dremio conventions."

The skill includes knowledge files covering Dremio CLI, Python SDK (dremioframe), SQL syntax, and REST API endpoints.

dremio-agent-md (Community)

The dremio-agent-md repository provides a master protocol file and browsable documentation sitemaps:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Reference it in your GEMINI.md:

## Dremio Documentation
- Read DREMIO_AGENT.md in ./dremio-agent-md/ for the Dremio protocol
- Use sitemaps in dremio_sitemaps/ to verify SQL syntax

This pairs well with Gemini CLI's Google Search grounding. The sitemaps provide structured offline references, while Search grounding provides real-time verification.

Approach 4: Build Your Own GEMINI.md Context

If the pre-built options do not fit your workflow, build a custom GEMINI.md tailored to your team's Dremio environment.

Create Project Context Files

.gemini/
  GEMINI.md              # Points to the reference files below
project-docs/
  dremio-conventions.md  # Team SQL rules
  table-schemas.md       # Exported schemas from Dremio
  common-queries.md      # Frequently used query patterns
  dremioframe-patterns.md # Python SDK code snippets

Write a Comprehensive GEMINI.md

# Team Dremio Context

## SQL Standards
- All tables are under the analytics namespace
- Bronze: analytics.bronze.*, Silver: analytics.silver.*, Gold: analytics.gold.*
- Always use TIMESTAMP, never DATE
- Validate function names against project-docs/dremio-conventions.md

## Authentication
- Use env var DREMIO_PAT for tokens
- Cloud endpoint: env var DREMIO_URI

## Reference Files
- SQL conventions: project-docs/dremio-conventions.md
- Table schemas (updated weekly): project-docs/table-schemas.md
- Common queries: project-docs/common-queries.md
- Python SDK patterns: project-docs/dremioframe-patterns.md

<PROTOCOL>
When writing Dremio SQL:
1. Read project-docs/table-schemas.md to verify table and column names
2. Read project-docs/dremio-conventions.md to validate function names
3. Use Google Search to verify any Dremio function not in the reference
</PROTOCOL>

The 1-million token context window means Gemini CLI can hold your entire schema reference, convention guide, and query library simultaneously without truncation.

Using Dremio with Gemini CLI: Practical Use Cases

Once Dremio is connected, Gemini CLI becomes a powerful data engineering partner in your terminal. Here are detailed examples.

Ask Natural Language Questions About Your Data

Ask Gemini CLI questions about your lakehouse in plain English:

"What were our top 10 customers by revenue last quarter? Show month-over-month trends and flag any with declining order frequency."

Gemini CLI uses the MCP connection to discover your tables, writes the SQL, runs it against Dremio, and returns formatted results with analysis. The 1-million token context window means it can hold large result sets and build on them across a session.

Follow up with multi-step analysis:

"For the customers with declining frequency, pull their support ticket history and calculate the correlation between ticket volume and order decline."

Gemini CLI maintains the full conversation context, including previous query results, and generates the follow-up query with cross-table joins. If it is unsure about a table name or column, it can use Google Search grounding to verify against live Dremio documentation.

Build a Locally Running Dashboard

Ask Gemini CLI to create a complete dashboard:

"Query our gold-layer sales views in Dremio and build a local HTML dashboard with Chart.js. Include monthly revenue trends, top products by region, and customer acquisition metrics. Make it filterable by date range and add a dark theme with print-to-PDF."

Gemini CLI will:

Use MCP to discover gold-layer views and their schemas
Write and execute SQL queries for each visualization
Generate an HTML file with embedded CSS, JavaScript, and Chart.js
Add interactive filter controls and export buttons
Save it to your project directory

Open the HTML file in a browser for a complete dashboard running from a local file. No server or deployment needed.

Create a Data Exploration App

Build an interactive tool:

"Create a Python Streamlit app that connects to Dremio using dremioframe. Include a schema browser sidebar with table counts, a data preview with pagination, a SQL query editor with syntax highlighting and execution, and CSV download. Generate requirements.txt and README."

Gemini CLI writes the full application:

app.py with Streamlit layout, dremioframe connection, and query execution
requirements.txt with pinned dependencies
.env.example with required environment variables
README.md with setup and run instructions

Run streamlit run app.py and your team has a local data explorer connected to the lakehouse.

Generate Data Pipeline Scripts

Automate data engineering workflows:

"Write a dremioframe script that implements a Medallion Architecture pipeline for our new product_events table. Bronze: ingest raw data with column renames and TIMESTAMP casts. Silver: deduplicate on event_id, validate required fields, apply business rules. Gold: aggregate daily active products, event counts by type, and conversion funnels. Include error handling, structured logging, and a dry-run mode."

Gemini CLI uses the GEMINI.md conventions and Dremio skill knowledge to produce production-quality pipeline code. Its Google Search grounding means it can verify Dremio function syntax in real time if the reference files do not cover a specific function.

Build API Endpoints Over Dremio Data

Create backend services:

"Build a FastAPI application that connects to Dremio using dremioframe. Create endpoints for customer segments, revenue by geography, and product performance trends. Include Pydantic response models, request validation, caching with TTL, and auto-generated OpenAPI docs."

Gemini CLI generates the complete API server with proper error handling and connection management. Deploy it locally with uvicorn main:app --reload or containerize for production.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time access
GEMINI.md	10 minutes	Convention enforcement, protocol blocks, Search grounding	Teams with specific SQL standards or project rules
Pre-Built Skills	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Quick start with broad coverage
Custom Context	30+ minutes	Tailored schemas, patterns, and team conventions	Mature teams with project-specific needs

Combine them for the strongest setup. The MCP server gives live data access; GEMINI.md enforces conventions with protocol blocks; pre-built skills provide broad Dremio knowledge; and custom context files capture your team's schemas and patterns.

Start with the MCP server for immediate value. Add a GEMINI.md with your SQL conventions. Use Google Search grounding as a safety net for syntax verification. As your team develops patterns, build out the context files with schemas and query libraries.

Get Started

Sign up for a free Dremio Cloud trial (30 days, $400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Add it to ~/.gemini/settings.json or .gemini/settings.json.
Clone dremio-agent-skill and tell Gemini CLI to read the skill.
Start a session and ask Gemini CLI to explore your Dremio catalog.

Dremio's Agentic Lakehouse gives Gemini CLI accurate data context: the semantic layer provides business meaning, query federation provides universal access, and Reflections provide interactive speed. Gemini CLI's massive context window holds it all, and Google Search grounding provides real-time verification as a safety net.

For more on the Dremio MCP Server, check out the official documentation or enroll in the free Dremio MCP Server course on Dremio University.

How to Use Dremio with Cursor: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 12:00:00 GMT

Cursor is an AI-native code editor built as a fork of VS Code. It integrates AI directly into the editing experience with features like Chat, Composer (multi-file editing), and inline code generation. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.

Connecting them gives Cursor's AI the context it needs to write accurate Dremio SQL, generate data pipeline code, and build applications against your lakehouse. Without this connection, Cursor treats Dremio like a generic database and guesses at function names and table paths. With it, the AI knows your schemas, your business logic encoded in views, and the correct Dremio SQL dialect.

Cursor's rules system is especially well-suited for Dremio integration. Rules files in .cursor/rules/ let you define granular, pattern-matched instructions that activate only when relevant. You can set Dremio SQL conventions to apply only when editing .sql files, and dremioframe patterns to apply only in Python files that import the SDK.

This post covers four approaches, ordered from quickest setup to most customizable.

Setting Up Cursor

If you do not already have Cursor installed:

Download Cursor from cursor.com (available for macOS, Linux, and Windows).
Install it by running the installer. Cursor replaces or runs alongside VS Code since it is a fork with the same extension ecosystem.
Sign in with a Cursor account. The free tier includes limited AI requests; Pro ($20/month) provides unlimited access.
Open a project by selecting File > Open Folder and pointing to your project directory.
Verify AI access by pressing Cmd+K (macOS) or Ctrl+K (Windows/Linux) to open the inline AI prompt. Type a question to confirm the AI is responding.

Cursor supports all VS Code extensions, themes, and keybindings. If you are migrating from VS Code, your existing setup transfers automatically.

Approach 1: Connect the Dremio Cloud MCP Server

The Model Context Protocol (MCP) is an open standard that lets AI tools call external services. Every Dremio Cloud project ships with a built-in MCP server. Cursor supports MCP natively through its settings panel.

For Claude-based tools like Claude Code, Dremio provides an official Claude plugin with guided setup. For Cursor, you configure the MCP connection through Cursor's built-in MCP settings.

Find Your Project's MCP Endpoint

Log into Dremio Cloud and open your project. Navigate to Project Settings > Info. The MCP server URL is listed on the project overview page. Copy it.

Set Up OAuth in Dremio Cloud

Dremio's hosted MCP server uses OAuth for authentication. Your existing access controls apply to every query the AI runs.

Go to Settings > Organization Settings > OAuth Applications.
Click Add Application and enter a name (e.g., "Cursor MCP").
Add the redirect URIs for Claude:
- https://claude.ai/api/mcp/auth_callback
- https://claude.com/api/mcp/auth_callback
Save and copy the Client ID.

Configure Cursor's MCP Connection

In Cursor, go to Settings > MCP. Click Add new MCP server and configure it with your Dremio project's MCP URL. You can also add the MCP server by creating a .cursor/mcp.json file in your project root:

{
  "mcpServers": {
    "dremio": {
      "url": "https://YOUR_PROJECT_MCP_URL",
      "auth": {
        "type": "oauth",
        "clientId": "YOUR_CLIENT_ID"
      }
    }
  }
}

Restart Cursor. The AI now has access to Dremio's MCP tools:

GetUsefulSystemTableNames returns an index of available tables with descriptions.
GetSchemaOfTable returns column names, types, and metadata.
GetDescriptionOfTableOrSchema pulls wiki descriptions and labels from the catalog.
GetTableOrViewLineage shows upstream data dependencies.
RunSqlQuery executes SQL and returns results as JSON.

Test the connection by opening Cursor Chat (Cmd+L) and asking: "What tables are available in Dremio?" The AI will call GetUsefulSystemTableNames and return your catalog contents.

Self-Hosted Alternative

For Dremio Software deployments, use the open-source dremio-mcp server. Clone the repo, configure it, then add it to Cursor's MCP settings:

git clone https://github.com/dremio/dremio-mcp
cd dremio-mcp
uv run dremio-mcp-server config create dremioai \
  --uri https://your-dremio-instance.com \
  --pat YOUR_PERSONAL_ACCESS_TOKEN

In .cursor/mcp.json:

{
  "mcpServers": {
    "dremio": {
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

The self-hosted server supports three modes: FOR_DATA_PATTERNS for data exploration (default), FOR_SELF for system analysis, and FOR_PROMETHEUS for correlating metrics with monitoring.

Approach 2: Use Cursor Rules for Dremio Context

Cursor's rules system is one of its strongest differentiators. Rules are markdown files in .cursor/rules/ that provide persistent AI instructions. Unlike a single monolithic context file, Cursor rules support pattern matching, so you can scope instructions to specific file types or directories.

Project-Wide Rules with .cursorrules

The simplest approach is a .cursorrules file in your project root. This loads into every AI interaction:

# Dremio SQL Conventions
- Use CREATE FOLDER IF NOT EXISTS (not CREATE NAMESPACE or CREATE SCHEMA)
- Tables in the Open Catalog use folder.subfolder.table_name without a catalog prefix
- External federated sources use source_name.schema.table_name
- Cast DATE to TIMESTAMP for consistent joins
- Use TIMESTAMPDIFF for duration calculations

# Credentials
- Never hardcode Personal Access Tokens. Use environment variable: DREMIO_PAT
- Dremio Cloud endpoint: environment variable DREMIO_URI

# Terminology
- Call it "Agentic Lakehouse", not "data warehouse"
- "Reflections" are pre-computed optimizations, not "materialized views"

Pattern-Matched Rules with .cursor/rules/

For more granular control, create rule files in .cursor/rules/ with .mdc (Markdown Cursor) extension. These files support YAML-like frontmatter that tells Cursor when to activate the rule:

---
description: Dremio SQL conventions for query files
globs: ["**/*.sql", "**/queries/**"]
alwaysApply: false
---

# Dremio SQL Rules

When writing or modifying SQL files for Dremio:
- Use CREATE FOLDER IF NOT EXISTS, never CREATE SCHEMA
- Validate function names against the Dremio SQL reference
- Use TIMESTAMPDIFF for duration calculations, not DATEDIFF
- Cast DATE columns to TIMESTAMP before joins
- Reference tables as folder.subfolder.table_name

Create a separate rule for Python SDK usage:

---
description: dremioframe Python SDK patterns
globs: ["**/*.py"]
alwaysApply: false
---

# dremioframe Conventions

When writing Python code that uses dremioframe:
- Import as: from dremioframe import DremioConnection
- Use environment variables for credentials: DREMIO_PAT, DREMIO_URI
- Always close connections in a finally block or use context managers
- For bulk operations, use df.to_dremio() with batch_size parameter

The globs field ensures these rules only activate when editing matching files. The alwaysApply: false setting means the AI loads them on demand rather than consuming context tokens on every interaction.

Referencing External Documentation

Keep rules files concise by pointing to reference documents:

---
description: Dremio documentation references
globs: ["**/*.sql", "**/*.py"]
alwaysApply: false
---

# Dremio Reference Docs
- For SQL syntax details, read `./docs/dremio-sql-reference.md`
- For Python SDK usage, read `./docs/dremioframe-guide.md`
- For REST API endpoints, read `./docs/dremio-rest-api.md`

Cursor loads the referenced files only when the AI needs them, keeping the context window efficient.

Approach 3: Install Pre-Built Dremio Skills and Docs

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill (Community)

The dremio-agent-skill repository contains a comprehensive skill directory with knowledge files and a .cursorrules file specifically designed for Cursor.

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cd dremio-agent-skill
./install.sh

Choose Local Project Install (Copy) to copy the .cursorrules file and knowledge directory into your project. The .cursorrules file provides Dremio conventions, and the knowledge/ directory contains detailed references for CLI, Python SDK, SQL syntax, and REST API.

After installation, Cursor automatically picks up the .cursorrules file and uses it for all AI interactions in the project.

dremio-agent-md (Community)

The dremio-agent-md repository provides a DREMIO_AGENT.md protocol file and browsable sitemaps of the Dremio documentation.

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Reference it in your .cursorrules:

For Dremio SQL validation, read DREMIO_AGENT.md in the dremio-agent-md directory.
Use the sitemaps in dremio_sitemaps/ to verify syntax before generating SQL.

Approach 4: Build Your Own Cursor Rules

If the pre-built options do not fit your workflow, create a custom rules setup tailored to your team's Dremio environment.

Create Rule Files

.cursor/rules/
  dremio-sql.mdc          # SQL conventions
  dremio-python.mdc       # dremioframe patterns
  dremio-schemas.mdc      # Team-specific table schemas
  dremio-api.mdc          # REST API patterns

Populate with Team Context

Export your actual table schemas from Dremio and save them as a rule:

---
description: Team Dremio table schemas
globs: ["**/*.sql", "**/*.py"]
alwaysApply: false
---

# Team Table Schemas

## analytics.gold.customer_metrics
- customer_id: VARCHAR (primary key)
- lifetime_value: DECIMAL(10,2)
- segment: VARCHAR (values: 'enterprise', 'mid-market', 'smb')
- last_order_date: TIMESTAMP
- churn_risk_score: FLOAT

## analytics.gold.revenue_daily
- date_key: TIMESTAMP
- product_category: VARCHAR
- region: VARCHAR
- revenue: DECIMAL(12,2)
- orders: INT

This gives Cursor exact schema knowledge for your project, so the AI generates SQL with correct column names and types instead of guessing.

Add Notepads for Reference Knowledge

Cursor also supports Notepads for longer reference documents. Create a notepad in .cursor/notepads/dremio-reference.md with comprehensive documentation. Notepads are available as @notepad references in Chat and Composer but do not auto-load, keeping your context efficient.

Using Dremio with Cursor: Practical Use Cases

Once Dremio is connected, Cursor becomes a powerful data development environment. Here are detailed examples.

Ask Natural Language Questions About Your Data

Open Cursor Chat (Cmd+L) and ask questions in plain English:

"What were our top 10 products by revenue last quarter? Break it down by region and show the trend."

Cursor uses the MCP connection to discover your tables, writes the SQL in the chat, and can run it against Dremio to return results. You get answers without switching to the Dremio UI.

Follow up with deeper analysis:

"Which of those top products has declining margins? Pull cost and revenue data for the last 6 months and show the margin trend."

Cursor maintains context across the chat session, building on previous results. This turns the editor into a conversational data analysis tool.

For teams with non-SQL users, Cursor Chat provides a natural language interface to the lakehouse directly inside the development environment.

Build a Locally Running Dashboard

Use Cursor Composer (Cmd+I) for multi-file generation:

"Query our gold-layer sales views in Dremio and build a local HTML dashboard with Chart.js. Include monthly revenue trends, top products by region, and customer acquisition metrics. Make it filterable by date range. Put the HTML, CSS, and JavaScript in separate files."

Cursor Composer will:

Use MCP to discover gold-layer views and their schemas
Create index.html with the dashboard layout
Create styles.css with the dark theme and responsive design
Create app.js with Chart.js configurations and data fetching
Embed query results as JSON data files
Add interactive filter controls

Open index.html in a browser for a complete dashboard running from local files. Cursor Composer excels at multi-file generation, making it ideal for this kind of project scaffolding.

Create a Data Exploration App

Build an interactive tool using Composer:

"Create a Streamlit app that connects to Dremio using dremioframe. Include a schema browser sidebar, a data preview tab with pagination, a SQL query editor with syntax highlighting, and CSV download buttons. Generate requirements.txt and a README."

Cursor generates the full application across multiple files:

app.py with Streamlit layout and dremioframe integration
requirements.txt with pinned dependencies
.env.example with required environment variables
README.md with setup and run instructions

Run pip install -r requirements.txt && streamlit run app.py for a local data explorer connected to your lakehouse.

Generate Data Pipeline Scripts

Automate data engineering with inline AI:

Highlight a comment in your Python file like # Create bronze-silver-gold pipeline for user_events table and press Cmd+K. Cursor generates the complete pipeline code inline:

Bronze: raw data ingestion with column renames and TIMESTAMP casts
Silver: deduplication, null checks, and type validation
Gold: business logic aggregations with CASE WHEN classifications
Error handling with retry logic
Structured logging for monitoring

The inline generation respects your .cursor/rules/ Dremio conventions, so the SQL follows your team's standards automatically.

Build API Endpoints Over Dremio Data

Use Composer to scaffold a REST API:

"Build a FastAPI application that connects to Dremio using dremioframe. Create endpoints for customer segments, revenue analytics, and product performance. Include Pydantic models, request validation, response caching, and auto-generated OpenAPI docs."

Cursor generates the complete API across multiple files with proper project structure, ready for uvicorn main:app --reload.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time access
Cursor Rules	15 minutes	Convention enforcement, pattern-matched context	Teams with specific SQL standards per file type
Pre-Built Skills	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Quick start with broad coverage
Custom Rules	30+ minutes	Tailored schemas, patterns, and team conventions	Mature teams with project-specific needs

Combine them for the strongest setup. The MCP server gives live data access; Cursor rules enforce conventions scoped to relevant file types; pre-built skills provide broad Dremio knowledge; and custom rules capture your team's specific schemas and patterns.

Start with the MCP server for immediate value. Add a .cursorrules file for project-wide conventions. As your team develops specific patterns, create .cursor/rules/*.mdc files with pattern matching for granular control.

Get Started

Sign up for a free Dremio Cloud trial (30 days, $400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Add it in Cursor's Settings > MCP or create .cursor/mcp.json.
Clone dremio-agent-skill and run ./install.sh with local project install.
Open Cursor Chat and ask it to explore your Dremio catalog.

Dremio's Agentic Lakehouse gives Cursor's AI accurate data context: the semantic layer provides business meaning, query federation provides universal access, and Reflections provide interactive speed. Cursor's rules system scopes that context intelligently, activating Dremio knowledge only when relevant.

For more on the Dremio MCP Server, check out the official documentation or enroll in the free Dremio MCP Server course on Dremio University.

How to Use Dremio with Claude CoWork: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 11:00:00 GMT

Claude CoWork is Anthropic's desktop agentic assistant. Unlike Claude Code (a terminal coding agent), CoWork operates as a general-purpose autonomous agent that reads and writes files, browses the web, manages tasks, and generates complete project artifacts. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections.

CoWork's strength is autonomous project execution. Give it a goal and grant it folder access, and it works through the steps independently. For data teams, this means CoWork can query your Dremio lakehouse, analyze the results, build a local dashboard, and write a summary report without you watching over every step.

The context mechanism in CoWork differs from code editors. There is no CLAUDE.md or AGENTS.md file. CoWork uses folder instructions and global instructions configured through the Claude Desktop app. This makes the integration approach different, but the end result is the same: an agent that understands your Dremio environment.

CoWork also has a unique advantage for Dremio users who are not developers. Because CoWork is a desktop assistant rather than a coding tool, analysts and business users can use it to ask natural language questions about their lakehouse data. The MCP connection handles the SQL generation and execution behind the scenes.

This post covers four approaches, from the quickest MCP connection to building a full Dremio knowledge folder.

Setting Up Claude CoWork

If you do not already have CoWork set up:

Download Claude Desktop from claude.ai/download (available for macOS and Windows).
Sign in with your Anthropic account (Pro, Team, or Enterprise subscription required for CoWork features).
Enable CoWork in the Claude Desktop app under Settings > Features > CoWork.
Grant folder access by clicking Add Folder and selecting your project directory.

CoWork operates as a desktop assistant, not a terminal tool. You interact with it through the Claude Desktop interface, describe tasks in natural language, and it autonomously reads files, writes code, browses the web, and generates project artifacts.

Approach 1: Connect the Dremio Cloud MCP Server

Every Dremio Cloud project includes a built-in MCP server. CoWork supports MCP through Claude Desktop's connector system.

Dremio also provides an official plugin for Claude that streamlines setup. If you use Claude Code alongside CoWork, you can install the plugin directly:

/plugin marketplace add dremio/claude-plugins
/plugin install dremio@dremio-plugins

Create a .env file with your credentials:

DREMIO_PAT=<your_personal_access_token>
DREMIO_PROJECT_ID=<your_project_id>

Then add the Dremio MCP server through the Claude web interface under Customize > Connectors > Add custom connector. CoWork automatically inherits MCP connections configured through the Claude web interface. Run /dremio-setup in Claude Code for step-by-step guidance.

Set Up OAuth in Dremio Cloud

Log into Dremio Cloud and go to Settings > Organization Settings > OAuth Applications.
Click Add Application and name it (e.g., "Claude CoWork").
Add the redirect URIs for Claude:
- https://claude.ai/api/mcp/auth_callback
- https://claude.com/api/mcp/auth_callback
Save and copy the Client ID.

Configure the MCP Connector

In Claude Desktop, open Settings > Connectors. Add a custom MCP connector with your Dremio project's MCP URL and the OAuth client ID. CoWork will now have access to Dremio's MCP tools:

GetUsefulSystemTableNames lists available tables.
GetSchemaOfTable returns column names and types.
GetDescriptionOfTableOrSchema pulls wiki descriptions from the catalog.
RunSqlQuery executes SQL and returns results.

Test it by telling CoWork: "Connect to Dremio and list the available tables in my project." The agent will use the MCP tools to browse your catalog.

Self-Hosted MCP

For Dremio Software, configure the open-source dremio-mcp server in Claude Desktop's claude_desktop_config.json:

{
  "mcpServers": {
    "dremio": {
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

Approach 2: Use Folder Instructions for Dremio Context

CoWork uses a folder-based context model. When you grant CoWork access to a folder, you can set instructions that apply whenever the agent works within that folder.

Setting Global Dremio Instructions

In Claude Desktop, go to Settings > CoWork > Global Instructions. Add Dremio conventions that apply to every task:

When working with Dremio:
- Use CREATE FOLDER IF NOT EXISTS, not CREATE NAMESPACE
- Tables in the Open Catalog use folder.subfolder.table_name without a catalog prefix
- External sources use source_name.schema.table_name
- Cast DATE to TIMESTAMP for consistent joins
- Never hardcode Personal Access Tokens; use environment variables
- Dremio is an Agentic Lakehouse, not a data warehouse

Setting Folder-Specific Instructions

When you grant CoWork access to a project folder, add instructions specific to that project:

This folder contains a Dremio analytics project.
- Read dremio-docs/sql-reference.md before writing any SQL
- All tables are under the analytics namespace
- Bronze: analytics.bronze.*, Silver: analytics.silver.*, Gold: analytics.gold.*
- Use environment variable DREMIO_PAT for authentication
- Use environment variable DREMIO_URI for the Dremio endpoint

Folder instructions load whenever CoWork operates in that directory, giving it project-specific context on top of the global Dremio defaults.

Agentic Memories

CoWork creates "agentic memories" as it works. After a few sessions with Dremio, CoWork builds persistent knowledge about your table schemas, common query patterns, and the SQL conventions it should follow. These memories survive across sessions, so the agent improves over time.

For example, after CoWork runs its first few Dremio queries in a project, it remembers which tables exist, which columns tend to be useful, and which SQL patterns work best. The next time you ask a question, CoWork draws on this accumulated knowledge to write better queries faster.

This is equivalent to CLAUDE.md or AGENTS.md but generated automatically rather than written by hand. For teams that do not want to maintain context files manually, agentic memories provide a self-improving alternative.

Approach 3: Load Pre-Built Dremio Docs into CoWork

Two community-supported open-source repositories provide Dremio context that CoWork can read directly.

Official vs. Community Resources: The dremio/claude-plugins plugin and the built-in Dremio Cloud MCP server are officially maintained by Dremio. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-md: Best Fit for CoWork (Community)

The dremio-agent-md repository is the best fit for CoWork. It contains DREMIO_AGENT.md (a master protocol file) and dremio_sitemaps/ (hierarchical documentation indices).

Clone it and grant CoWork access to the folder:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Set the folder instructions to:

Before answering any Dremio questions, read DREMIO_AGENT.md in this folder.
Use the sitemaps in dremio_sitemaps/ to verify SQL syntax and find documentation.

CoWork will read the protocol file, learn the SQL conventions, and use the sitemaps to validate any Dremio queries it generates.

dremio-agent-skill: Knowledge Files (Community)

The dremio-agent-skill repository provides a broader set of knowledge files covering CLI, Python SDK, SQL, and REST API:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill

Grant CoWork access to this folder and set instructions to: "Read dremio-skill/SKILL.md for Dremio capabilities. Reference the knowledge/ directory for SQL syntax, REST API, and Python SDK documentation."

Approach 4: Build a Custom Dremio Knowledge Folder

Create a dedicated folder with everything CoWork needs for your Dremio project:

dremio-context/
  README.md               # Overview and instructions
  sql-conventions.md       # Team SQL rules
  table-schemas.md         # Exported schemas from Dremio
  common-queries.md        # Frequently used query patterns
  dremioframe-examples.md  # Python SDK code snippets
  rest-api-patterns.md     # API call examples

Write a README.md that tells CoWork how to use the folder:

# Dremio Project Context

Read this folder to understand our Dremio setup before working on data tasks.

## Quick Reference
- SQL conventions: sql-conventions.md
- Table schemas: table-schemas.md (updated weekly)
- Common queries: common-queries.md
- Python SDK: dremioframe-examples.md
- REST API: rest-api-patterns.md

## Rules
- Always use CREATE FOLDER IF NOT EXISTS
- Use TIMESTAMPDIFF for duration calculations
- Credentials are in environment variables, never hardcoded

Grant CoWork access to this folder and set folder instructions to: "Before any Dremio task, read README.md in the dremio-context folder."

Export your actual table schemas from Dremio regularly and update table-schemas.md. Include the queries your team runs most often in common-queries.md. This grows into a living knowledge base that CoWork uses to generate increasingly accurate output.

Using Dremio with CoWork: Practical Use Cases

Once Dremio is connected, CoWork can execute complete data projects autonomously. Here are detailed examples.

Ask Natural Language Questions About Your Data

Ask CoWork plain questions about your data:

"What were our top 10 products by revenue last quarter? Break it down by region."

CoWork uses the MCP connection to discover relevant tables, writes the SQL, runs the query against Dremio, and returns formatted results with analysis. No SQL knowledge required on your part.

Take it further with follow-up questions:

"Which of those top products had the highest return rates? Pull the return reasons and show the most common issues."

CoWork remembers the previous results and builds on them. Its agentic memory system stores what it learns about your tables, so subsequent questions in the same project get faster, more accurate answers.

This pattern is especially valuable for non-technical users. Business analysts, product managers, and executives can use CoWork to query the lakehouse without learning SQL or navigating the Dremio UI.

Build Locally Running Dashboards

Tell CoWork to build a complete dashboard:

"Query our gold-layer sales views in Dremio, then build me a local HTML dashboard with charts showing monthly revenue trends, top customers, and regional breakdowns. Use Chart.js for the visualizations. Add date filters and a dark theme."

CoWork will:

Use MCP to discover gold-layer views and their schemas
Write and execute SQL queries for each visualization
Generate an HTML file with embedded CSS, JavaScript, and Chart.js
Add interactive filter controls for date range and region
Save it to your project folder

Open the HTML file in a browser, and you have a working dashboard running entirely from a local file. Share it with stakeholders by dropping it in Slack or email. No server or deployment needed.

For recurring dashboards, tell CoWork to regenerate it weekly. Its agentic memory remembers the queries and file structure from the previous run.

Create Data Exploration Apps

Ask CoWork to build a more interactive tool:

"Create a Python Flask app that connects to Dremio using dremioframe. It should let me type a table name and see the schema, preview 100 rows, and run custom SQL queries. Include a clean UI with syntax highlighting and CSV download."

CoWork writes the Python code, creates the HTML templates, and generates a requirements.txt. Run pip install -r requirements.txt && python app.py and you have a local data exploration app connected to your lakehouse.

This is especially useful for teams who need quick internal tools without going through a formal development cycle.

Generate Automated Reports

Schedule CoWork to generate recurring analytical reports:

"Query this week's data quality metrics from Dremio's gold layer, compare them to last week, and write a markdown report with tables and recommendations. Include row count trends, null percentages by column, and any columns that exceeded the 5% null threshold. Save it to the reports/ folder."

CoWork runs the queries, computes the comparisons, generates a formatted report with tables, and writes recommendations based on the data. The report is ready to share with stakeholders without any manual analysis.

Build API Endpoints Over Dremio Data

Create backend services that serve lakehouse data:

"Build a FastAPI application that queries Dremio's gold-layer views and serves customer analytics through REST endpoints. Add endpoints for customer segments, revenue by geography, and cohort retention. Include request validation and JSON response formatting."

CoWork generates the full application with proper error handling and dremioframe connection management. Deploy it locally or containerize it for production.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Connector	5 minutes	Live queries, schema browsing, catalog access	Natural language data exploration, ad-hoc analysis
Folder Instructions	10 minutes	Convention enforcement, project context	Teams with specific SQL standards
Pre-Built Docs	5 minutes	Comprehensive Dremio knowledge	Quick setup with broad coverage
Custom Knowledge Folder	30+ minutes	Tailored schemas, queries, and patterns	Mature teams with specific data models

Start with the MCP connector. It gives CoWork live data access in five minutes, and you can immediately start asking natural language questions. Add folder instructions and knowledge files as you develop team-specific conventions.

Get Started

Sign up for Dremio Cloud free for 30 days ($400 in compute credits).
Set up OAuth and configure the MCP connector in Claude Desktop.
Clone dremio-agent-md and grant CoWork folder access.
Ask CoWork to explore your Dremio catalog.
Try: "Query my sales data in Dremio and build a local dashboard with Chart.js."

Dremio's Agentic Lakehouse gives CoWork the data foundation it needs: the semantic layer provides business context, query federation provides universal data access, and Reflections provide interactive speed. CoWork's autonomous execution model turns that data access into complete deliverables, from dashboards to reports to data apps.

For more on the Dremio MCP Server, see the official documentation or take the free Dremio MCP Server course on Dremio University.

How to Use Dremio with Claude Code: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 10:00:00 GMT

Claude Code is Anthropic's terminal-based coding agent. It reads your files, writes code, runs commands, and maintains context across a session. Dremio is a unified lakehouse platform that gives AI agents three things they need to answer business questions accurately: deep business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.

Connecting them means your coding agent can query live data, validate SQL against real schemas, and generate scripts that actually work against your lakehouse. Without this connection, Claude Code treats Dremio like any other database and often hallucinates function names or syntax. With it, the agent knows your table schemas, your business logic encoded in views, and the correct Dremio SQL dialect.

This post covers four approaches, ordered from quickest setup to most customizable.

Setting Up Claude Code

If you do not already have Claude Code installed, here is how to get started:

Install Node.js (version 18 or later) from nodejs.org.

Install Claude Code globally via npm:

npm install -g @anthropic-ai/claude-code

Launch Claude Code by running claude in your terminal from any project directory.
Authenticate with your Anthropic API key or Claude Pro/Team subscription on first launch.

Claude Code runs in your terminal and reads your project files for context. It can execute shell commands, edit files, and interact with MCP servers. No IDE or editor is required.

Approach 1: Connect the Dremio Cloud MCP Server

The Model Context Protocol (MCP) is an open standard that lets AI tools call external services. Every Dremio Cloud project ships with a built-in MCP server. Claude Code supports MCP natively. Connecting them takes about five minutes.

The fastest path is the official Dremio plugin for Claude Code from the dremio/claude-plugins repository. This is maintained by Dremio and provides guided setup.

Find Your Project's MCP Endpoint

Log into Dremio Cloud and open your project. Navigate to Project Settings > Info. The MCP server URL is listed on the project overview page. Copy it.

Set Up OAuth in Dremio Cloud

Dremio's hosted MCP server uses OAuth for authentication. This means Claude Code connects with your identity and your existing access controls apply to every query the agent runs.

Go to Settings > Organization Settings > OAuth Applications.
Click Add Application.
Enter an application name (e.g., "Claude Code MCP").
Add the redirect URIs for Claude:
- https://claude.ai/api/mcp/auth_callback
- https://claude.com/api/mcp/auth_callback
Save the application and copy the Client ID.

Configure Claude Code's MCP Client

Claude Code reads MCP server definitions from a .mcp.json file. Create one in your project root:

{
  "mcpServers": {
    "dremio": {
      "url": "https://YOUR_PROJECT_MCP_URL",
      "auth": {
        "type": "oauth",
        "clientId": "YOUR_CLIENT_ID"
      }
    }
  }
}

For a global configuration that applies across all your projects, place the file at ~/.mcp.json instead.

Restart Claude Code. The agent now has access to Dremio's MCP tools:

GetUsefulSystemTableNames returns an index of available tables with descriptions.
GetSchemaOfTable returns column names, types, and metadata for any table or view.
GetDescriptionOfTableOrSchema pulls wiki descriptions and labels you have set in the Dremio catalog.
GetTableOrViewLineage shows upstream dependencies.
RunSqlQuery executes SQL and returns results as JSON.

You can verify the connection by asking Claude Code: "What tables are available in Dremio?" The agent will call GetUsefulSystemTableNames and return your catalog contents.

Official Dremio Plugin for Claude Code

Dremio provides an official Claude Code plugin that streamlines setup. Install it from the plugin marketplace:

/plugin marketplace add dremio/claude-plugins
/plugin install dremio@dremio-plugins

Create a .env file in your project directory with your credentials:

DREMIO_PAT=<your_personal_access_token>
DREMIO_PROJECT_ID=<your_project_id>

Then add the Dremio MCP server through the Claude web interface under Customize > Connectors > Add custom connector. Claude Code automatically inherits the connection.

Run /dremio-setup in Claude Code for step-by-step guidance. The plugin walks you through OAuth configuration, including setting the redirect URI to http://localhost/callback,https://claude.ai/api/mcp/auth_callback.

This is the recommended starting point for Claude Code users because it is officially maintained by Dremio and handles the configuration details for you.

Self-Hosted Alternative

If you run Dremio Software instead of Dremio Cloud, use the open-source dremio-mcp repository:

git clone https://github.com/dremio/dremio-mcp
cd dremio-mcp
uv run dremio-mcp-server config create dremioai \
  --uri https://your-dremio-instance.com \
  --pat YOUR_PERSONAL_ACCESS_TOKEN
uv run dremio-mcp-server config create claude

The second command writes the MCP server entry directly into Claude's desktop config. For Claude Code (terminal), add the server to your .mcp.json:

{
  "mcpServers": {
    "dremio": {
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

The self-hosted server supports three modes:

FOR_DATA_PATTERNS for exploring and querying data (default)
FOR_SELF for system introspection and performance analysis
FOR_PROMETHEUS for correlating Dremio metrics with Prometheus

Approach 2: Use CLAUDE.md for Dremio Context

MCP gives Claude Code live access to your data. But sometimes you need the agent to follow specific conventions, use the right SQL dialect, or know where to find documentation. The MCP connection tells Claude Code what data exists. Context files tell it how your team works with that data.

What CLAUDE.md Does

Claude Code auto-loads CLAUDE.md from your project root at the start of every session. It acts as persistent instructions that survive across conversations. You do not need to re-explain your project every time you start a new session.

The file supports three placement levels. A global ~/.claude/CLAUDE.md applies to every project you open. A project-root CLAUDE.md applies to that specific repo. And .claude/rules/*.md files let you split rules into focused modules that are loaded with the same priority. Project-level files override global ones, so you can set organizational defaults and override them per-repo.

Writing a Dremio-Focused CLAUDE.md

Here is an example CLAUDE.md that teaches Claude Code how to work with Dremio:

# Project Context

This project uses Dremio Cloud as its lakehouse platform.

## Dremio SQL Conventions
- Use `CREATE FOLDER IF NOT EXISTS` (not CREATE NAMESPACE or CREATE SCHEMA)
- Tables in the built-in Open Catalog use `folder.subfolder.table_name` without a catalog prefix
- External federated sources use `source_name.schema.table_name`
- Cast DATE columns to TIMESTAMP for consistent joins
- Use TIMESTAMPDIFF for duration calculations

## Credentials
- Never hardcode Personal Access Tokens. Use environment variable: DREMIO_PAT
- Dremio Cloud endpoint is in environment variable: DREMIO_URI

## API Reference
- REST API docs: https://docs.dremio.com/current/reference/api/
- SQL reference: https://docs.dremio.com/current/reference/sql/
- For detailed SQL validation, read ./dremio-docs/sql-reference.md

## Terminology
- Call it "Agentic Lakehouse", not "data warehouse"
- "Reflections" are pre-computed optimizations, not "materialized views"
- "Open Catalog" is built on Apache Polaris
- The AI Agent is a co-pilot, not a chatbot

Progressive Disclosure with Supplemental Files

Keep CLAUDE.md under 300 lines. For detailed references, store them in separate files and tell Claude Code where to find them:

## Documentation References
- For Dremio SQL syntax details, read `./docs/dremio-sql-reference.md`
- For Python SDK (dremioframe) usage, read `./docs/dremioframe-guide.md`
- For REST API endpoints, read `./docs/dremio-rest-api.md`

Claude Code only loads these files when it needs them, keeping your context window efficient. You can also instruct the agent explicitly: "Before writing any Dremio SQL, read ./docs/dremio-sql-reference.md to verify syntax."

You can also place rule files in .claude/rules/ and they will be auto-loaded with the same priority as CLAUDE.md. This is useful for separating concerns. For example, .claude/rules/dremio-conventions.md for SQL rules and .claude/rules/project-style.md for code style.

Approach 3: Install Pre-Built Dremio Skills and Docs

Beyond the official plugin, two community-supported open-source repositories provide ready-made Dremio context for coding agents. Both work with Claude Code.

Official vs. Community Resources: The dremio/claude-plugins plugin is officially maintained by Dremio. The repositories below are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product. Libraries like dremioframe (the Dremio Python SDK referenced in the skill) are also community-supported.

dremio-agent-skill: Full Agent Skill (Community)

The dremio-agent-skill repository contains a complete skill directory that teaches AI assistants how to interact with Dremio.

What is included:

dremio-skill/
  SKILL.md          # Entry point defining capabilities
  knowledge/        # Comprehensive docs for:
    cli/            #   Dremio CLI administration
    python/         #   dremioframe Python SDK
    sql/            #   SQL syntax, Iceberg DML, metadata
    rest-api/       #   REST API endpoints
  rules/
    .cursorrules    # Config for Cursor/VS Code
    AGENTS.md       # Config for OpenCode/Codex

Installation:

Run the interactive installer:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cd dremio-agent-skill
./install.sh

The installer asks you to choose:

Global Install (Symlink) symlinks the skill to ~/.claude/skills/ so every Claude Code session discovers it automatically. Updates to the cloned repo are reflected immediately.
Local Project Install (Copy) copies the skill into your project directory and sets up .claude symlinks so Claude Code auto-detects it. The skill travels with your repo, so every team member gets the same context.

After installation, start Claude Code and try: "Using the Dremio skill, write a dremioframe script to query my customer table."

dremio-agent-md: Documentation Protocol (Community)

The dremio-agent-md repository takes a different approach. Instead of a skill with structured knowledge files, it provides a master protocol file and a browsable sitemap of the entire Dremio documentation.

What is included:

DREMIO_AGENT.md defines how the agent should validate SQL, handle security (credentials via .env), and navigate the documentation.
dremio_sitemaps/ contains hierarchical markdown indices of official Dremio docs for both Cloud and Software versions.

Usage with Claude Code:

Clone the repo into your project or a reference directory:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Then tell Claude Code at the start of your session:

"Read DREMIO_AGENT.md in the dremio-agent-md directory to understand Dremio protocols. Use the sitemaps in dremio_sitemaps/ to verify any Dremio features or SQL syntax before generating code."

The agent will navigate the sitemaps to find the correct documentation page for whatever feature you are working with, like looking up the right function signature before writing a query.

This approach is especially useful when you need Claude Code to validate SQL against the official docs rather than rely on its training data.

Approach 4: Build Your Own Dremio Skill

If the pre-built options do not cover your specific workflow, build a custom skill. A skill is just a directory with a SKILL.md file and optional supporting docs.

Create the Skill Directory

my-dremio-skill/
  SKILL.md
  knowledge/
    sql-conventions.md
    rest-api-endpoints.md
    project-schemas.md

Write SKILL.md

The SKILL.md file needs YAML frontmatter for discovery and markdown instructions for the agent:

---
name: My Dremio Skill
description: Custom conventions and API patterns for our team's Dremio Cloud project
---

# My Dremio Skill

## When to Use
Use this skill when working with Dremio queries, dremioframe scripts,
or any code that interacts with our lakehouse.

## SQL Rules
- All tables live under the `analytics` namespace
- Use `analytics.bronze.*` for raw views, `analytics.silver.*` for joins,
  `analytics.gold.*` for final datasets
- Always use TIMESTAMP, never DATE
- Validate function names against `knowledge/sql-conventions.md`

## Authentication
- Use environment variable DREMIO_PAT for Personal Access Tokens
- Cloud endpoint: Use environment variable DREMIO_URI

## Reference Files
- SQL conventions: knowledge/sql-conventions.md
- REST API: knowledge/rest-api-endpoints.md
- Project schemas: knowledge/project-schemas.md

Install the Skill

For Claude Code, place the skill in one of these locations:

Global: ~/.claude/skills/my-dremio-skill/
Project-local: .claude/skills/my-dremio-skill/

Claude Code discovers skills by reading their SKILL.md files. When a user prompt matches the skill description, the agent loads the full instructions automatically.

Add Knowledge Files

Populate the knowledge/ directory with the specific references your team needs. You might include:

Your project's table schemas exported from Dremio
SQL patterns that are specific to your data model
dremioframe code snippets for common operations
REST API call examples with your specific endpoints

The advantage of a custom skill over a generic CLAUDE.md is discoverability. Skills are loaded on demand based on semantic matching, so they do not consume context tokens until they are needed.

Using Dremio with Claude Code: Practical Use Cases

Once Dremio is connected, Claude Code becomes a data engineering partner. Here are detailed examples you can try immediately.

Ask Natural Language Questions About Your Data

The simplest and most powerful use case. Ask Claude Code questions in plain English and get answers from production data:

"What were our top 10 customers by revenue last quarter? Show month-over-month trends."

Claude Code uses the MCP connection to discover your tables, writes the SQL, runs it against Dremio, and returns formatted results with analysis. You get answers from production data in seconds without writing a single query yourself.

You can go deeper with follow-up questions:

"Which of those top 10 customers had declining order frequency? Pull their last 6 months of order data and calculate the trend."

Because Claude Code maintains context across the session, it remembers the previous query results and builds on them. The MCP connection gives it live access to run the follow-up query without you needing to re-explain the schema.

This pattern turns Claude Code into a conversational analytics tool. Business analysts who are comfortable with English but not SQL can use it to explore data, test hypotheses, and generate insights directly from the lakehouse.

Build a Locally Running Dashboard

Ask Claude Code to create a complete, self-contained dashboard:

"Query our gold-layer sales views in Dremio and build a local HTML dashboard with Chart.js. Include monthly revenue trends, top products by region, and customer acquisition metrics. Make it filterable by date range and downloadable as PDF."

Claude Code will:

Use MCP to discover your gold-layer views and their schemas
Write and execute SQL queries to pull the relevant data
Generate an HTML file with embedded CSS, JavaScript, and Chart.js configurations
Embed the query results directly into the JavaScript as data arrays
Add interactive filters and a print-to-PDF button
Save everything to your project directory

Open the HTML file in a browser and you have an interactive dashboard running from a local file. No server, no deployment, no infrastructure. Share it with your team by dropping it in Slack or email.

For recurring dashboards, save the prompt in a script and re-run it weekly to regenerate the dashboard with fresh data from Dremio.

Create a Data Exploration App

Build a more interactive tool for ongoing data exploration:

"Create a Python Streamlit app that uses dremioframe to connect to Dremio. Include a schema browser sidebar, a data preview tab with pagination, and a SQL query editor with results. Add download buttons for CSV export and a query history panel."

Claude Code writes the full Python application:

app.py with Streamlit layout, dremioframe connection, and query execution
requirements.txt with pinned dependencies
.env.example showing required environment variables
README.md with setup instructions

Run pip install -r requirements.txt && streamlit run app.py and you have a local data exploration tool connected to your lakehouse. Your team can use it for ad-hoc analysis without needing direct access to the Dremio UI.

This pattern works well for creating internal tools quickly. Instead of waiting for a formal BI tool deployment, you can have a working data explorer in minutes.

Generate Data Pipeline Scripts

Automate data engineering workflows:

"Write a dremioframe script that reads new CSV files from the staging folder, creates bronze views in Dremio, builds silver views with data quality validations (null checks, type casting, deduplication), and creates gold views with business logic aggregations. Include error handling, logging, and a dry-run mode."

Claude Code uses the Dremio skill to write production-quality pipeline code that follows Medallion Architecture conventions. The script includes:

Bronze layer: raw data ingestion with column renames and TIMESTAMP casts
Silver layer: data quality rules, deduplication, and join logic
Gold layer: business metric aggregations and CASE WHEN classifications
Error handling with retry logic for transient Dremio connection issues
Structured logging for pipeline monitoring

Build API Endpoints Over Dremio Data

Create a REST API that serves lakehouse data to other applications:

"Build a FastAPI application that connects to Dremio using dremioframe. Create endpoints for: GET /api/customers (paginated), GET /api/customers/{id}/orders, GET /api/analytics/revenue?period=monthly. Add request validation, error handling, and OpenAPI documentation."

Claude Code generates a complete API server with typed request/response models, query parameterization to prevent SQL injection, and auto-generated Swagger docs. Deploy it locally or containerize it for production use.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time data access
CLAUDE.md	10 minutes	Convention enforcement, doc references, credential rules	Teams with specific SQL standards or project conventions
Pre-Built Skills	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Getting started quickly with broad Dremio coverage
Custom Skill	30+ minutes	Tailored to your exact schemas, patterns, and workflows	Mature teams with project-specific conventions

These approaches are not mutually exclusive. A common setup combines the MCP server for live data access with a custom CLAUDE.md for project conventions. Or start with the pre-built dremio-agent-skill and add a CLAUDE.md for your team-specific overrides.

The strongest configuration uses all four layers: MCP for live connectivity, CLAUDE.md for project rules, a pre-built skill for general Dremio knowledge, and custom knowledge files for your specific schemas and patterns.

If you are evaluating Dremio for the first time, start with the MCP server alone. It takes five minutes and gives you immediate value. As your usage matures and you need the agent to follow team conventions or validate against specific documentation, layer in the context files and skills.

Get Started

Sign up for a free Dremio Cloud trial (30 days, $400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Add it to Claude Code's .mcp.json.
Clone dremio-agent-skill and run ./install.sh.
Start Claude Code and ask it to explore your Dremio catalog.

Dremio's Agentic Lakehouse gives Claude Code what it needs to write accurate SQL: the semantic layer provides business context, query federation provides universal data access, and Reflections provide interactive speed. The MCP server is the bridge that connects them.

For more on the Dremio MCP Server, check out the official documentation or enroll in the free Dremio MCP Server course on Dremio University.

How to Use Dremio with Amazon Kiro: Connect, Query, and Build Data Apps

Alex Merced — Thu, 05 Mar 2026 09:00:00 GMT

Amazon Kiro is an agentic AI IDE from AWS that introduces spec-driven development to the coding workflow. Instead of jumping straight to code, Kiro helps you define structured specifications — requirements, technical designs, and task breakdowns — before writing a single line. It then generates code that follows those specs and keeps everything in sync as the project evolves. Dremio is a unified lakehouse platform that provides business context through its semantic layer, universal data access through query federation, and interactive speed through Reflections and Apache Arrow.

Connecting them gives Kiro's agent the context it needs to write accurate Dremio SQL, generate data pipelines, and build applications against your lakehouse. Kiro's spec-driven approach is especially well-suited for data projects: you can define your data model requirements in plain language, let Kiro generate the technical design, and then have it build the implementation with full traceability back to the original requirements.

Kiro's hooks system adds event-driven automation, so documentation, tests, and validation can update automatically as your Dremio code changes.

This post covers four approaches, ordered from quickest setup to most customizable.

Setting Up Amazon Kiro

If you do not already have Kiro installed:

Download Kiro from kiro.dev (available for macOS, Linux, and Windows).
Sign in with your AWS account, Google account, or GitHub account.
Open a project by selecting File > Open Folder and pointing to your project directory.
Explore the interface — Kiro includes a file explorer, an AI chat panel, a specs panel for viewing requirements/design/tasks, and a hooks panel for event-driven automations.

Kiro is built on the VS Code platform, so existing VS Code extensions and themes are compatible. It is free to use during the preview period.

Approach 1: Connect the Dremio Cloud MCP Server

Every Dremio Cloud project ships with a built-in MCP server. Kiro supports MCP natively and integrates deeply with AWS MCP servers.

For Claude-based tools, Dremio provides an official Claude plugin with guided setup. For Kiro, you configure the MCP connection through the IDE settings or project configuration.

Find Your Project's MCP Endpoint

Log into Dremio Cloud and navigate to Project Settings > Info. Copy the MCP server URL.

Set Up OAuth in Dremio Cloud

Go to Settings > Organization Settings > OAuth Applications.
Click Add Application and enter a name (e.g., "Kiro MCP").
Add the appropriate redirect URIs.
Save and copy the Client ID.

Configure Kiro's MCP Connection

In Kiro, open the MCP settings and add a new server. You can configure via the settings UI or create a .kiro/mcp.json file:

{
  "mcpServers": {
    "dremio": {
      "url": "https://YOUR_PROJECT_MCP_URL"
    }
  }
}

Kiro now has access to Dremio's MCP tools:

GetUsefulSystemTableNames returns available tables.
GetSchemaOfTable returns column definitions.
GetDescriptionOfTableOrSchema pulls catalog descriptions.
GetTableOrViewLineage shows data lineage.
RunSqlQuery executes SQL and returns results.

Test by asking the AI chat: "What tables are available in Dremio?"

Kiro Powers

Kiro supports "Powers" — curated bundles of MCP servers, steering files, and hooks for specific development workflows. If an AWS or community Dremio Power becomes available, you can install it from the Powers panel to get a pre-configured Dremio development environment.

Self-Hosted Alternative

For Dremio Software deployments, configure the dremio-mcp server:

{
  "mcpServers": {
    "dremio": {
      "command": "uv",
      "args": [
        "run", "--directory", "/path/to/dremio-mcp",
        "dremio-mcp-server", "run"
      ]
    }
  }
}

Approach 2: Use Kiro Specs for Dremio Context

Kiro's spec-driven development is its most distinctive feature. Instead of free-form context files, Kiro uses structured specification documents that the AI generates and maintains.

Generating Specs for a Dremio Project

Tell Kiro to create specs for your data project:

"I need a data analytics pipeline that reads from Dremio's lakehouse, transforms the data using a Medallion Architecture, and serves results through a REST API."

Kiro generates three spec files in .kiro/specs/:

requirements.md — User stories in structured format:

1. As a data engineer, I want to ingest raw data from Dremio bronze tables
   so that I can process it through the pipeline.
2. As a data analyst, I want cleaned data in gold views
   so that I can run accurate business queries.
3. As an application developer, I want REST endpoints over gold data
   so that I can build dashboards and reports.

design.md — Technical design covering architecture, data flow, table schemas, and technology choices.

tasks.md — A breakdown of implementation tasks that Kiro tracks as you build.

Adding Dremio Conventions to Specs

You can refine the generated specs with Dremio-specific conventions:

"Update the design to use Dremio SQL conventions: CREATE FOLDER IF NOT EXISTS, folder.subfolder.table_name paths, TIMESTAMPDIFF for durations. Use dremioframe for Python connections and environment variables for credentials."

Kiro updates the design.md and tasks.md to reflect these conventions. All code generated from these specs will follow the conventions automatically.

Steering Files

Kiro also supports steering files — markdown documents that provide persistent context similar to .cursorrules or CLAUDE.md. Create a .kiro/steering/dremio.md file:

# Dremio Conventions

## SQL
- Use CREATE FOLDER IF NOT EXISTS
- Tables: folder.subfolder.table_name
- Cast DATE to TIMESTAMP for joins
- Use TIMESTAMPDIFF for durations

## Credentials
- DREMIO_PAT and DREMIO_URI from environment variables
- Never hardcode tokens

## Terminology
- "Agentic Lakehouse" not "data warehouse"
- "Reflections" not "materialized views"

Approach 3: Install Pre-Built Dremio Skills and Docs

Official vs. Community Resources: Dremio provides an official plugin for Claude Code users and the built-in Dremio Cloud MCP server is an official Dremio product. The repositories below, along with libraries like dremioframe, are community-supported projects from the Dremio Developer Advocacy team. They are actively maintained but not part of the core Dremio product.

dremio-agent-skill (Community)

The dremio-agent-skill repository provides knowledge files:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-skill
cd dremio-agent-skill
./install.sh

Copy the knowledge directory into your project and reference it in Kiro's steering files.

dremio-agent-md (Community)

The dremio-agent-md repository provides documentation sitemaps:

git clone https://github.com/developer-advocacy-dremio/dremio-agent-md

Reference it in .kiro/steering/dremio.md:

For SQL validation, read DREMIO_AGENT.md in ./dremio-agent-md/.

Approach 4: Build Custom Specs and Hooks

Kiro's hooks system offers a unique approach to maintaining data project consistency.

Creating Dremio Hooks

Hooks are event-driven automations that trigger when files change. Create hooks that automatically validate Dremio SQL:

On SQL file save — A hook that validates SQL syntax against Dremio conventions:

"Create a hook that triggers when any .sql file is saved. It should read the file, validate that it uses CREATE FOLDER IF NOT EXISTS instead of CREATE SCHEMA, checks for proper table path formatting, and flags any deprecated function names."

On pipeline code change — A hook that updates tests:

"Create a hook that triggers when any Python file in the pipelines/ directory changes. It should update the corresponding test file to match the new pipeline logic, using dremioframe mocking patterns."

Hooks keep your Dremio project self-maintaining. As code changes, documentation and tests update automatically.

Custom Steering Files

Create comprehensive steering files in .kiro/steering/:

.kiro/steering/
  dremio-sql.md        # SQL conventions
  dremio-python.md     # dremioframe patterns
  dremio-schemas.md    # Team table schemas
  dremio-pipeline.md   # Pipeline architecture rules

These files are loaded into every Kiro interaction and ensure consistent code generation.

Using Dremio with Kiro: Practical Use Cases

Once Dremio is connected, Kiro's spec-driven approach creates traceable, well-documented data projects.

Ask Natural Language Questions About Your Data

In the chat panel:

"What were our top 10 products by revenue last quarter? Show growth rates and compare to the same period last year."

Kiro uses MCP to discover tables, writes SQL, and returns results. Unlike other tools, Kiro can also generate a spec that documents the analysis methodology for reproducibility.

Follow up:

"For the declining products, pull customer sentiment from support tickets. Is there a correlation between product issues and revenue decline?"

Kiro tracks the analytical thread and can generate a formal analysis spec for the investigation.

Build a Locally Running Dashboard

Start with specs:

"I need a self-contained HTML dashboard showing Dremio gold-layer metrics: revenue trends, customer acquisition, and regional performance. Spec it out first, then build it."

Kiro generates the requirements, design, and tasks first, then builds the dashboard following the specs. Every file traces back to a requirement, making it easy to review and maintain.

Create a Data Exploration App

Spec-driven app development:

"Spec and build a Streamlit app connected to Dremio via dremioframe. Requirements: schema browsing, SQL query editor, data preview, CSV export. Generate all files."

Kiro creates the full spec, then generates the application. The tasks.md tracks progress, and hooks can keep tests updated as you iterate.

Generate Data Pipeline Scripts

Spec-driven data engineering:

"Spec a Medallion Architecture pipeline for product_events. Requirements: bronze ingestion, silver cleaning, gold aggregation. Design should use dremioframe and follow our SQL conventions. Then implement it."

Kiro generates the full spec suite (requirements, design, tasks), then writes the pipeline code. Every transformation traces back to a requirement, and hooks validate the SQL on every save.

Build API Endpoints Over Dremio Data

Spec-driven API development:

"Spec and build a FastAPI service over Dremio gold-layer views. Requirements: customer analytics, revenue data, product metrics. Design should include Pydantic models and caching."

Kiro generates the complete API with full traceability to the requirements.

Which Approach Should You Use?

Approach	Setup Time	What You Get	Best For
MCP Server	5 minutes	Live queries, schema browsing, catalog exploration	Data analysis, SQL generation, real-time access
Kiro Specs	15 minutes	Structured requirements, design, traceable implementation	Teams valuing documentation and traceability
Pre-Built Skills	5 minutes	Comprehensive Dremio knowledge (CLI, SDK, SQL, API)	Quick start with broad coverage
Custom Hooks	30+ minutes	Event-driven validation, auto-updating tests and docs	Mature teams with CI-like automation needs

Start with the MCP server for live data access. Use Kiro's spec-driven flow for any project beyond a quick query. Add hooks for automated validation as your project matures.

Get Started

Sign up for a free Dremio Cloud trial (30 days, $400 in compute credits).
Find your project's MCP endpoint in Project Settings > Info.
Add it in Kiro's MCP settings or .kiro/mcp.json.
Tell Kiro to generate specs for your Dremio data project.
Let Kiro build the implementation from the specs.

Dremio's Agentic Lakehouse gives Kiro accurate data context, and Kiro's spec-driven methodology ensures every line of generated code traces back to a documented requirement. This is especially valuable for data engineering, where auditability and traceability matter.

For more on the Dremio MCP Server, check out the official documentation or enroll in the free Dremio MCP Server course on Dremio University.

Connect Dremio Software to Dremio Cloud: Hybrid Federation Across Deployments

Alex Merced — Mon, 02 Mar 2026 05:00:00 GMT

Dremio Cloud can connect to Dremio Software (self-managed) instances as a federated data source. This creates a hybrid deployment where Dremio Cloud serves as the primary query interface while accessing datasets managed by Dremio Software instances running in your own data centers or private cloud.

This connector is designed for organizations that have existing Dremio Software deployments and are adopting Dremio Cloud for new workloads, or that need to federate data across a cloud-managed Dremio platform and on-premises Dremio instances.

Why Connect Dremio Software to Dremio Cloud

Hybrid Federation

Your Dremio Software instance manages on-premises data sources — Oracle databases, SQL Server, network-attached file storage, and internal data lakes. Dremio Cloud manages cloud-native sources — S3, BigQuery, Snowflake, and cloud-hosted databases. By connecting Dremio Software to Dremio Cloud, you can write a single SQL query that joins on-premises data (through Dremio Software) with cloud data (through Dremio Cloud).

-- Join on-premises data via Dremio Software with cloud data in Dremio Cloud
SELECT
  cloud.customer_name,
  cloud.cloud_revenue,
  onprem.erp_balance,
  onprem.last_payment_date,
  CASE
    WHEN cloud.cloud_revenue > 100000 AND onprem.erp_balance < 5000 THEN 'Good Standing'
    WHEN onprem.erp_balance > 50000 THEN 'At Risk'
    ELSE 'Standard'
  END AS account_health
FROM analytics.gold.cloud_customers cloud
JOIN "dremio-onprem".onprem.erp_accounts onprem ON cloud.customer_id = onprem.customer_id
ORDER BY cloud.cloud_revenue DESC;

Incremental Cloud Migration

Organizations don't shut down on-premises data centers overnight. Connecting Dremio Software to Dremio Cloud lets you:

Start using Dremio Cloud for new cloud-native workloads
Continue using Dremio Software for on-premises sources
Federate across both from a single Dremio Cloud interface
Gradually migrate data sources from Software to Cloud as on-premises systems are decommissioned

Consolidated Governance

Users access both on-premises and cloud data through Dremio Cloud's interface. Dremio Cloud's governance policies (column masking, row-level filtering) apply to the federated view of data, providing a single governance layer across all data.

Prerequisites

Dremio Software instance accessible from Dremio Cloud over HTTPS
- Version 24.0 or later recommended
- Arrow Flight endpoint enabled and accessible (port 32010 or 443 with TLS)
Authentication: Username/password or Personal Access Token for the Dremio Software instance
Network: The Dremio Software instance must be reachable from Dremio Cloud's network. Options:
- Public endpoint with TLS
- VPN/VPC peering
- AWS PrivateLink or equivalent
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Dremio Software to Dremio Cloud

1. Add the Source

Click "+" in the Dremio Cloud console and select Dremio from the source types.

2. Configure Connection

Name: Descriptive identifier (e.g., dremio-onprem or datacenter-west).
Host: The hostname or IP address of your Dremio Software coordinator node.
Port: Arrow Flight port (typically 32010, or 443 with TLS).
SSL/TLS: Enable if the Software instance uses encrypted connections.

3. Set Authentication

Provide credentials for a Dremio Software user account. Consider creating a dedicated service account with appropriate permissions:

Read access to the virtual datasets (views) and physical datasets you want to federate
User impersonation support if you want Dremio Cloud queries to execute as the requesting user on Dremio Software

4. User Impersonation

User impersonation allows Dremio Cloud to pass the identity of the requesting user to Dremio Software. When enabled, queries executed through Dremio Cloud run with the permissions of the authenticated user on the Dremio Software side. This preserves your existing Dremio Software access control policies.

Without impersonation, all Cloud queries execute as the service account configured in the connection, which may have broader access than individual users should.

5. Configure Advanced Settings

Set Reflection Refresh, Metadata refresh intervals, and connection properties. Click Save.

Querying Across Deployments

-- Query on-premises data through Dremio Software
SELECT
  department,
  employee_count,
  avg_salary
FROM "dremio-onprem".hr.department_summary;

-- Join on-premises HR data with cloud-native analytics
SELECT
  d.department,
  d.employee_count,
  d.avg_salary,
  c.department_cloud_spend,
  ROUND(c.department_cloud_spend / d.employee_count, 2) AS cloud_cost_per_employee
FROM "dremio-onprem".hr.department_summary d
JOIN analytics.gold.cloud_infrastructure_costs c ON d.department = c.department
ORDER BY cloud_cost_per_employee DESC;

Build a Semantic Layer Across Deployments

CREATE VIEW analytics.gold.enterprise_360 AS
SELECT
  onprem.employee_id,
  onprem.employee_name,
  onprem.department,
  onprem.office_location,
  cloud.cloud_account_id,
  cloud.monthly_cloud_spend,
  CASE
    WHEN cloud.monthly_cloud_spend > 10000 THEN 'Heavy Cloud User'
    WHEN cloud.monthly_cloud_spend > 1000 THEN 'Moderate'
    ELSE 'Light'
  END AS cloud_usage_tier
FROM "dremio-onprem".hr.employees onprem
LEFT JOIN analytics.gold.cloud_accounts cloud ON onprem.employee_id = cloud.owner_id;

In the Catalog, click Edit → Generate Wiki and Generate Tags.

AI-Powered Analytics Across Deployments

Dremio AI Agent

The AI Agent lets users ask questions spanning both on-premises and cloud data: "Which departments have the highest cloud cost per employee?" or "Show me heavy cloud users in the engineering department." The Agent reads your semantic layer's wiki descriptions and generates SQL that joins across both Dremio deployments.

Dremio MCP Server

Connect Claude or ChatGPT to your federated data:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs for your AI client
Connect via mcp.dremio.cloud/mcp/{project_id}

A CTO asks Claude "Compare cloud infrastructure costs per department with on-premises headcount" and gets insights spanning both deployment models.

AI SQL Functions

-- Classify departments by cloud optimization potential
SELECT
  department,
  employee_count,
  cloud_cost_per_employee,
  AI_CLASSIFY(
    'Based on cloud spending patterns, classify optimization potential',
    'Department: ' || department || ', Employees: ' || CAST(employee_count AS VARCHAR) || ', Cloud Cost/Employee: $' || CAST(cloud_cost_per_employee AS VARCHAR),
    ARRAY['Well Optimized', 'Room for Improvement', 'Over-Provisioned', 'Needs Audit']
  ) AS optimization_status
FROM (
  SELECT
    d.department,
    d.employee_count,
    ROUND(c.department_cloud_spend / d.employee_count, 2) AS cloud_cost_per_employee
  FROM "dremio-onprem".hr.department_summary d
  JOIN analytics.gold.cloud_infrastructure_costs c ON d.department = c.department
);

Important Considerations

Network Latency

Cross-network queries between Dremio Cloud and on-premises Dremio Software add network latency. Optimize by:

Using Reflections to cache frequently accessed on-premises data in Dremio Cloud
Creating aggregated views on the Dremio Software side that pre-compute common metrics — transfer summarized data rather than raw tables
Minimizing the amount of raw data transferred across the network

Cloud Egress Costs

Data returned from Dremio Software to Dremio Cloud may incur cloud egress charges if the Software instance runs in a different network or cloud provider. Strategies to minimize egress:

Build pre-aggregated views on the Software side
Use Reflections to cache results (data transfers once per refresh, not per query)
Filter data as close to the source as possible

Version Compatibility

Keep Dremio Software at version 24.0 or later for best compatibility with Dremio Cloud. Older versions may have limited feature support through the federation connector.

Security

Enable TLS for all connections between Dremio Cloud and Software
Use a dedicated service account with minimal necessary permissions
Enable user impersonation for proper access control propagation
Consider network-level security (VPN, PrivateLink) for on-premises connections

Monitoring and Troubleshooting

Monitor the health and performance of your hybrid deployment:

Query profiles: Use Dremio Cloud's query profiler to identify slow cross-deployment queries. Look for high data transfer volumes that suggest missing Reflections.
Metadata refresh timing: If Dremio Cloud shows stale schema from Dremio Software, decrease the metadata refresh interval.
Connection pool management: For high-concurrency workloads, monitor connection usage between Cloud and Software. Increase the maximum idle connections if you see connection timeout errors.
Latency benchmarks: Establish baseline latency for cross-deployment queries. If latency degrades, check network connectivity and consider adding Reflections to cache frequently accessed data.

Track these metrics to ensure your hybrid architecture delivers consistent performance as usage grows.

Governance Across Deployments

Dremio Cloud's Fine-Grained Access Control (FGAC) applies governance to the federated view of data:

Column masking: Mask sensitive on-premises fields (employee SSN, salary) from specific Cloud user roles
Row-level filtering: Regional Cloud users see only their region's data from on-premises sources
Unified policies: Same governance rules apply whether data comes from the Software instance, Cloud sources, or external databases

Connect BI Tools via Arrow Flight

BI tools connected to Dremio Cloud via Arrow Flight access both cloud and on-premises data through a single connection:

Tableau: Dremio connector — one connection serves data from both deployments
Power BI: Dremio ODBC driver or native connector
Python/Pandas: pyarrow.flight client for programmatic access
dbt: dbt-dremio adapter for transformation workflows

All queries benefit from Reflections, governance, and the semantic layer — regardless of where the source data resides.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration enables developers to query federated data from their IDE. Ask Copilot "Compare cloud costs per department with on-premises headcount" and it generates SQL using your semantic layer that spans both deployments.

Reflections for Hybrid Optimization

Create Reflections on hybrid views to cache cross-deployment query results:

Build views that join Cloud and Software data
Create Reflections on those views
Set refresh intervals based on how frequently the underlying on-premises data changes

After creation, dashboard queries that span both deployments are served from Dremio Cloud's Reflection cache — eliminating network latency for repeat queries.

Migration Planning: Software to Cloud

Use the Dremio-to-Dremio connector as a migration bridge:

Phase 1 — Federation: Connect Dremio Software to Dremio Cloud. All existing Software views remain accessible from Cloud.
Phase 2 — Parallel Development: Build new views and Reflections in Dremio Cloud while continuing to maintain Software views.
Phase 3 — Source Migration: Gradually move individual data sources (PostgreSQL, Oracle, S3) from Software connections to Cloud connections. Update views to reference Cloud-native sources.
Phase 4 — Decommission: Once all sources are connected to Cloud, remove the Dremio Software connection.

During the migration, users experience no disruption — they continue querying through Dremio Cloud while the underlying sources are being transitioned.

Common Deployment Architectures

Hub-and-Spoke Model

Dremio Cloud serves as the central hub, with multiple Dremio Software instances as spokes. Each spoke manages a specific data center or business unit:

Spoke A: Finance data center (Oracle, SQL Server, DB2)
Spoke B: Manufacturing data center (SAP HANA, PostgreSQL)
Spoke C: Research data center (S3, MongoDB)

Dremio Cloud federates across all spokes, providing a single analytics interface for the entire organization.

Staged Migration Model

For organizations migrating to the cloud in waves:

Wave 1: Non-sensitive workloads migrate to Cloud with direct source connections
Wave 2: Sensitive workloads use Software as a proxy (governance-compliant data access)
Wave 3: Remaining workloads migrate as regulatory and security requirements are met

Disaster Recovery Model

Dremio Software serves as a fallback if Cloud connectivity is temporarily unavailable. On-premises critical workloads run against Software; Cloud handles all other analytics. This architecture provides business continuity for mission-critical dashboards and reports.

Performance Best Practices

Maximize hybrid performance with these strategies:

Pre-aggregate on Software side: Build views in Dremio Software that SUM, COUNT, and AVG at the granularity Cloud queries need. Transfer megabytes, not gigabytes.
Use Reflections aggressively: Create Reflections on every cross-deployment view. Network latency disappears once results are cached.
Schedule Reflection refreshes strategically: Refresh during off-peak hours when network bandwidth is available.
Monitor data transfer volumes: Use Dremio Cloud's query profiler to identify queries that transfer large volumes from Software. Convert these to Reflections first.

Get Started

Organizations can seamlessly federate across Dremio deployments, enable AI analytics on combined on-premises and cloud data, and migrate incrementally to the cloud — all while maintaining unified governance. The Dremio-to-Dremio connector is the bridge that makes hybrid lakehouse analytics practical.

Whether you're running a single Dremio Software instance in one data center or managing multiple Software installations across global facilities, Dremio Cloud provides a unified analytical interface. Combine the raw data processing power of on-premises Dremio Software with the AI capabilities, Reflections, and managed infrastructure of Dremio Cloud. The result is a truly hybrid analytics platform that grows with your cloud migration at whatever pace your organization requires. No rip-and-replace, no big-bang migration — just a gradual, governed transition that protects your existing investments.

Try Dremio Cloud free for 30 days and connect your existing Dremio Software instances.

Dremio's Built-in Open Catalog: Your Zero-Configuration Apache Iceberg Lakehouse

Alex Merced — Mon, 02 Mar 2026 04:00:00 GMT

Every Dremio Cloud account starts with a built-in Open Catalog — a fully managed Apache Iceberg catalog with integrated storage. When you create a Dremio Cloud project, you immediately have a catalog where you can create namespaces (folders), tables, and views without connecting any external sources, configuring storage, or setting up credentials.

This isn't a bare-bones starting point. The built-in Open Catalog is a production-grade Iceberg catalog with automated performance management, Autonomous Reflections, time travel, branching, and full DML support. It's the fastest path from "sign up" to "running analytics."

Organizations typically spend days or weeks setting up external catalogs — provisioning S3 buckets, configuring IAM roles, debugging credential chains, and testing connectivity. With the built-in Open Catalog, you skip all of that. Your first CREATE TABLE runs minutes after account creation.

The Open Catalog is particularly powerful for teams adopting a lakehouse architecture for the first time. Instead of evaluating AWS Glue, Unity Catalog, and Snowflake Open Catalog (each with different setup complexity, vendor dependencies, and pricing models), start with the built-in catalog. You can always connect external catalogs later and federate across them.

Cross-Catalog Federation

The Open Catalog works alongside external catalogs. A common architecture:

Open Catalog: Dremio-created analytical tables and views (gold layer, semantic layer)
AWS Glue: Existing Iceberg tables managed by Spark/EMR pipelines
PostgreSQL: Operational application data

Dremio federates across all three in a single SQL query. Views in the Open Catalog can reference tables from any connected source, creating a unified analytical layer that spans your entire data estate.

Branching and Tagging

The Open Catalog supports Iceberg's branching and tagging capabilities:

Branches: Create isolated copies of table metadata for development and testing. Changes on a branch don't affect the main table until merged.
Tags: Create named snapshots for milestone tracking (e.g., quarterly-report-2024-Q2).

These features enable data engineering workflows where teams can test transformations on branches before promoting changes to production tables.

Why Start with the Built-in Open Catalog

Zero Configuration

External catalogs (Glue, Unity, Snowflake Open Catalog) require AWS IAM roles, network configuration, credential management, and catalog-specific setup. The built-in Open Catalog requires nothing — it's already configured when your project is created. Create a folder, write SQL, and start working.

Automated Performance Management

Dremio automatically manages the performance of Iceberg tables in the built-in catalog:

Auto-compaction: Small files are automatically merged into optimally sized files (typically 256MB). This prevents the "small file problem" that degrades query performance over time.
Manifest rewriting: Table manifests are automatically optimized for faster metadata reads.
Data clustering: Dremio sorts data based on query patterns to improve predicate pushdown efficiency.
Vacuuming: Expired snapshots and orphaned data files are automatically cleaned up.
Results caching: Query results are cached and served for identical subsequent queries.

None of these require manual OPTIMIZE commands or scheduled maintenance jobs. Dremio handles it all in the background.

Autonomous Reflections

For tables in the built-in catalog, Dremio can automatically create and manage Reflections based on observed query patterns. If a specific view is queried frequently with certain filters and aggregations, Dremio creates a Reflection to accelerate those patterns without any manual configuration. This automated acceleration means your most common queries get faster over time.

Time Travel

Query any table as it existed at any point in the past:

-- Query table as it was 7 days ago
SELECT * FROM catalog_folder.my_table AT TIMESTAMP '2024-06-01 00:00:00';

-- Query a specific snapshot
SELECT * FROM catalog_folder.my_table AT SNAPSHOT '1234567890123456789';

Time travel is valuable for auditing ("What did customer balances look like at quarter end?"), debugging ("What changed in the last 24 hours?"), and compliance ("Show me the data as it was on the regulatory reporting date").

Full DML Support

The built-in catalog supports all standard DML operations:

-- INSERT
INSERT INTO analytics.bronze.events
SELECT event_type, user_id, event_timestamp
FROM "s3-datalake".events.raw_events
WHERE event_date = CURRENT_DATE - INTERVAL '1' DAY;

-- UPDATE
UPDATE analytics.silver.customers
SET segment = 'Enterprise'
WHERE total_spend > 100000;

-- DELETE
DELETE FROM analytics.bronze.events
WHERE event_timestamp < CURRENT_DATE - INTERVAL '365' DAY;

-- MERGE (upsert)
MERGE INTO analytics.silver.customers AS target
USING (
  SELECT customer_id, SUM(amount) AS total_spend
  FROM analytics.bronze.orders
  GROUP BY customer_id
) AS source
ON target.customer_id = source.customer_id
WHEN MATCHED THEN UPDATE SET total_spend = source.total_spend
WHEN NOT MATCHED THEN INSERT (customer_id, total_spend) VALUES (source.customer_id, source.total_spend);

Getting Started: Create Your First Tables

When you query items in the built-in catalog, you don't include a source name prefix — just the folder path and table/view name:

-- Create namespace structure
CREATE FOLDER IF NOT EXISTS analytics;
CREATE FOLDER IF NOT EXISTS analytics.bronze;
CREATE FOLDER IF NOT EXISTS analytics.silver;
CREATE FOLDER IF NOT EXISTS analytics.gold;

-- Create a table from an external source
CREATE TABLE analytics.bronze.raw_orders AS
SELECT order_id, customer_id, product_id, quantity, price, order_date
FROM "postgres-orders".public.orders
WHERE order_date >= '2024-01-01';

-- Create a transformed table
CREATE TABLE analytics.silver.enriched_orders AS
SELECT
  o.order_id,
  o.customer_id,
  c.customer_name,
  c.region,
  o.product_id,
  p.product_name,
  p.category,
  o.quantity,
  o.price,
  o.quantity * o.price AS total_amount,
  o.order_date
FROM analytics.bronze.raw_orders o
JOIN "postgres-orders".public.customers c ON o.customer_id = c.customer_id
JOIN "postgres-orders".public.products p ON o.product_id = p.product_id;

-- Create an analytics view
CREATE VIEW analytics.gold.revenue_summary AS
SELECT
  region,
  category,
  DATE_TRUNC('month', order_date) AS month,
  SUM(total_amount) AS revenue,
  COUNT(*) AS orders,
  COUNT(DISTINCT customer_id) AS unique_customers,
  ROUND(SUM(total_amount) / COUNT(DISTINCT customer_id), 2) AS revenue_per_customer
FROM analytics.silver.enriched_orders
GROUP BY region, category, DATE_TRUNC('month', order_date);

Build a Semantic Layer

CREATE VIEW analytics.gold.product_performance AS
SELECT
  category,
  product_name,
  SUM(total_amount) AS revenue,
  COUNT(*) AS orders,
  CASE
    WHEN SUM(total_amount) > 100000 THEN 'Top Performer'
    WHEN SUM(total_amount) > 10000 THEN 'Solid'
    ELSE 'Emerging'
  END AS performance_tier
FROM analytics.silver.enriched_orders
GROUP BY category, product_name;

In the Catalog, click Edit (pencil icon) → Generate Wiki and Generate Tags. These descriptions power AI features.

AI-Powered Analytics

Dremio AI Agent

The AI Agent reads your semantic layer to answer questions in plain English: "What's our top performing product category this quarter?" or "Show me revenue per customer by region." The wiki descriptions you create tell the Agent what "top performing" and "revenue per customer" mean, generating accurate SQL automatically.

Dremio MCP Server

The Dremio MCP Server connects external AI tools to your catalog data:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs for your AI client (Claude, ChatGPT)
Connect via mcp.dremio.cloud/mcp/{project_id}

A VP of Product asks Claude "Compare our product category performance and identify emerging categories with high growth potential" and gets governed, accurate analysis.

AI SQL Functions

-- Generate product analysis with AI
SELECT
  product_name,
  performance_tier,
  revenue,
  AI_GENERATE(
    'Write a one-sentence growth strategy for this product',
    'Product: ' || product_name || ', Category: ' || category || ', Revenue: $' || CAST(revenue AS VARCHAR) || ', Tier: ' || performance_tier
  ) AS growth_strategy
FROM analytics.gold.product_performance;

-- Classify products for portfolio management
SELECT
  product_name,
  AI_CLASSIFY(
    'Based on revenue and order volume, classify investment priority',
    'Revenue: $' || CAST(revenue AS VARCHAR) || ', Orders: ' || CAST(orders AS VARCHAR),
    ARRAY['Strategic Investment', 'Maintain', 'Optimize', 'Sunset']
  ) AS investment_priority
FROM analytics.gold.product_performance;

Built-in vs. External Catalogs

Feature	Built-in Open Catalog	External Catalogs (Glue, Unity, etc.)
Setup	Zero configuration	Requires IAM, networking, credentials
Auto-compaction	✅ Automatic	✅ For Iceberg tables
Autonomous Reflections	✅ Automatic	Manual Reflections only
Time travel	✅ Full support	✅ For Iceberg tables
Write support	✅ Full DML	Varies by catalog
Credential management	None needed	IAM roles or keys required
Storage costs	Included in Dremio	Separate cloud storage costs

The built-in catalog is ideal for getting started, prototyping, and production workloads. External catalogs are valuable when your organization already manages data in Glue, Unity, or Snowflake Open Catalog and wants to query that data through Dremio.

Governance in the Open Catalog

Dremio's Fine-Grained Access Control (FGAC) provides enterprise-grade governance on all Open Catalog data:

Column masking: Mask sensitive columns (customer PII, financial details) from specific roles. A data analyst sees customer_name but not social_security_number.
Row-level filtering: Automatically restrict data visibility based on user roles. A regional manager querying revenue_summary sees only their region.
Unified policies: The same governance policies apply across Open Catalog tables, external catalogs, and database sources — one set of rules for all data.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent queries, and MCP Server interactions.

Connect BI Tools via Arrow Flight

Dremio's Arrow Flight connector provides 10-100x faster data transfer than JDBC/ODBC. After building views in the Open Catalog:

Tableau: Use the Dremio connector for direct Arrow Flight access
Power BI: Use Dremio's ODBC driver or native connector
Python/Pandas: Use pyarrow.flight for high-speed programmatic data access
Looker: Connect via JDBC
dbt: Use dbt-dremio adapter for SQL-based transformations

All queries benefit from Autonomous Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Open Catalog data from their IDE. Ask Copilot "Show me this week's revenue by product category" and it generates SQL using your semantic layer — without switching to the Dremio console.

Data Lifecycle Management

The Open Catalog supports a complete data lifecycle:

Bronze layer: Ingest raw data from external sources using CREATE TABLE ... AS SELECT
Silver layer: Apply transformations, deduplication, and type casting with CREATE TABLE ... AS SELECT or MERGE for incremental updates
Gold layer: Create analytical views with business logic for the semantic layer
Archival: Use DELETE with time-based conditions to remove old data; use time travel to access historical snapshots before deletion

This medallion architecture runs entirely within Dremio — no external ETL tools, Spark clusters, or scheduled scripts needed.

Incremental Loading Patterns

For ongoing data ingestion, use MERGE to incrementally update tables without full reloads:

-- Incremental merge: only update changed records
MERGE INTO analytics.silver.customers AS target
USING (
  SELECT customer_id, customer_name, email, segment, updated_at
  FROM "postgres-crm".public.customers
  WHERE updated_at > (SELECT MAX(updated_at) FROM analytics.silver.customers)
) AS source
ON target.customer_id = source.customer_id
WHEN MATCHED THEN UPDATE SET
  customer_name = source.customer_name,
  email = source.email,
  segment = source.segment,
  updated_at = source.updated_at
WHEN NOT MATCHED THEN INSERT (customer_id, customer_name, email, segment, updated_at)
  VALUES (source.customer_id, source.customer_name, source.email, source.segment, source.updated_at);

This pattern transfers only changed records, minimizing network traffic and compute costs.

Time Travel Best Practices

Time travel is particularly valuable in the Open Catalog for:

End-of-quarter reporting: Query tables at exact quarter-end timestamps for regulatory submissions
Debugging data issues: Compare current data with a previous snapshot to identify when and what changed
Audit trails: Demonstrate data state at any point in time for compliance requirements
Recovery: If a bad UPDATE or DELETE corrupts data, query the pre-change snapshot and restore

-- Compare current vs 24-hours-ago to find changed records
SELECT current_data.customer_id, current_data.segment AS new_segment, old_data.segment AS old_segment
FROM analytics.silver.customers current_data
JOIN analytics.silver.customers AT TIMESTAMP '2024-06-14 00:00:00' old_data
  ON current_data.customer_id = old_data.customer_id
WHERE current_data.segment <> old_data.segment;

Get Started

Every Dremio Cloud account includes the Open Catalog, ready to go. No setup, no configuration, no external dependencies. Create your first table in under a minute.

The Open Catalog isn't just for prototyping — it's production-grade from day one. Organizations run terabyte-scale analytical workloads on the built-in catalog with automated performance management handling compaction, vacuuming, and Reflection optimization in the background. Start small with a few tables, then scale to hundreds of tables and dozens of users as your lakehouse grows. The same zero-configuration promise holds at scale.

For teams new to the lakehouse concept, the Open Catalog is the lowest-friction entry point available. Data engineers familiar with SQL can build a complete medallion architecture (bronze → silver → gold) in a single afternoon, with AI capabilities and governance ready to activate immediately.

Connect Any Iceberg REST Catalog to Dremio Cloud: Universal Lakehouse Access

Alex Merced — Mon, 02 Mar 2026 03:00:00 GMT

The Apache Iceberg REST Catalog specification defines a standard HTTP API for managing Iceberg table metadata. Any catalog implementation that conforms to this specification — Apache Polaris, Amazon S3 Tables, Confluent Tableflow, Tabular, Apache Gravitino, and custom-built services — can connect to Dremio Cloud through a single connector type.

This is the most flexible catalog connector Dremio offers. Instead of needing a purpose-built connector for every catalog vendor, the Iceberg REST Catalog connector works with any compliant implementation. As new catalogs emerge — and they're emerging rapidly in the open lakehouse ecosystem — this connector ensures Dremio supports them from day one.

The Iceberg REST specification is becoming the universal standard for lakehouse catalog interoperability. AWS launched Amazon S3 Tables (a fully managed Iceberg catalog with REST API) in late 2024, Confluent released Tableflow for streaming-to-Iceberg ingestion, and Apache Gravitino provides multi-catalog governance. All of these work with Dremio's REST Catalog connector without any Dremio-side code changes.

Credential Vending Advantage

Many REST catalogs support credential vending — the ability to issue temporary, scoped storage credentials to clients. When configured, Dremio receives short-lived tokens that grant access only to the specific data files needed for a query. This eliminates the need to store long-lived S3 access keys or Azure storage keys in Dremio's connection configuration, significantly reducing the security surface area. One REST catalog connection replaces what would otherwise require separate storage credentials for every S3 bucket, Azure container, or GCS bucket containing your Iceberg tables.

Why Iceberg REST Catalog Users Need Dremio

Universal Compatibility

The Iceberg REST Catalog connector works with any catalog implementation that conforms to the Iceberg REST API spec. This includes:

Catalog	Type	Credential Vending
Apache Polaris	Open source	Yes
Amazon S3 Tables	AWS managed	Yes
Confluent Tableflow	Confluent managed	Yes
Tabular	SaaS	Yes
Apache Gravitino	Open source	Varies
Custom REST implementations	Self-hosted	Varies

You're not locked into specific catalog vendors. Deploy Apache Polaris today, consider S3 Tables tomorrow — the same Dremio connector works for both.

Read and Write Support

Dremio supports full DML (INSERT, UPDATE, DELETE, MERGE) on Iceberg tables managed by REST catalogs. You can create tables, run transformations, build data pipelines, and maintain your lakehouse entirely through Dremio's SQL engine. No need for separate Spark clusters or ETL jobs for routine operations.

Multi-Catalog Federation

Connect multiple REST catalogs alongside databases (PostgreSQL, MySQL, Oracle), object storage (S3, Azure), cloud warehouses (Snowflake, BigQuery), and other catalogs (Glue, Unity) — then query across all of them in a single SQL statement.

Automated Iceberg Maintenance

Dremio automatically compacts small files, rewrites manifests for faster metadata reads, and clusters data based on query patterns — even for tables managed by external REST catalogs.

Multiple Authentication Methods

The connector supports Bearer Token, OAuth 2.0 (client credentials flow), and custom authentication headers, accommodating the security requirements of different catalog implementations.

Flexible Storage Credential Management

Some REST catalogs vend temporary storage credentials (short-lived S3/Azure/GCS tokens) for reading and writing data files. Dremio supports credential vending where available. When a catalog doesn't vend credentials, you can configure storage access directly in Dremio.

Prerequisites

REST Catalog endpoint URL — the base URL of the catalog API (e.g., https://my-polaris.example.com/api/catalog)
Authentication credentials — Bearer token, OAuth client ID/secret, or custom headers depending on the catalog
Storage access — either through credential vending (catalog provides temporary tokens) or direct storage credentials (S3, Azure, GCS)
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect to Dremio Cloud

1. Add the Source

Click "+" and select Iceberg REST Catalog.

2. Configure Connection Details

Name: Descriptive identifier (e.g., polaris-catalog or s3-tables).
Catalog Endpoint URL: The base URL for the REST API.

3. Set Authentication

Choose from:

Bearer Token: For token-based authentication (e.g., PAT tokens).
OAuth 2.0: Client ID and client secret for OAuth client credentials flow.
None: For catalogs that use other authentication methods (configured via custom headers).

4. Configure Storage

If credential vending is supported, Dremio receives temporary credentials automatically. Otherwise, configure S3 (access key/secret or IAM role), Azure (shared key or service principal), or GCS (service account key) credentials.

5. Advanced Settings

Custom Headers: Additional HTTP headers required by the catalog.
Query Parameters: URL parameters appended to catalog API requests.
Catalog-specific properties: Key-value pairs for vendor-specific configuration.

6. Set Reflection and Metadata Refresh, then Save

Query and Write to REST Catalog Tables

-- Query Iceberg tables from a REST catalog
SELECT order_id, customer_id, order_total, order_date
FROM "rest-catalog".ecommerce.orders
WHERE order_date >= '2024-01-01' AND order_total > 100
ORDER BY order_total DESC;

-- Write to the catalog
INSERT INTO "rest-catalog".analytics.daily_summary
SELECT
  DATE_TRUNC('day', order_date) AS day,
  COUNT(*) AS order_count,
  SUM(order_total) AS revenue,
  AVG(order_total) AS avg_order_value
FROM "rest-catalog".ecommerce.orders
WHERE order_date = CURRENT_DATE - INTERVAL '1' DAY
GROUP BY 1;

-- MERGE for upserts
MERGE INTO "rest-catalog".analytics.customer_metrics AS target
USING (
  SELECT customer_id, COUNT(*) AS orders, SUM(order_total) AS total_spent
  FROM "rest-catalog".ecommerce.orders
  WHERE order_date >= CURRENT_DATE - INTERVAL '30' DAY
  GROUP BY customer_id
) AS source
ON target.customer_id = source.customer_id
WHEN MATCHED THEN UPDATE SET orders = source.orders, total_spent = source.total_spent
WHEN NOT MATCHED THEN INSERT (customer_id, orders, total_spent) VALUES (source.customer_id, source.orders, source.total_spent);

Federate with Other Sources

-- Join REST catalog data with PostgreSQL and S3
SELECT
  rc.order_id,
  rc.order_total,
  pg.customer_name,
  pg.region,
  s3.support_tickets
FROM "rest-catalog".ecommerce.orders rc
JOIN "postgres-crm".public.customers pg ON rc.customer_id = pg.customer_id
LEFT JOIN "s3-support".tickets.customer_counts s3 ON rc.customer_id = s3.customer_id
WHERE rc.order_total > 500
ORDER BY rc.order_total DESC;

Build a Semantic Layer

CREATE VIEW analytics.gold.customer_value AS
SELECT
  rc.customer_id,
  pg.customer_name,
  pg.region,
  SUM(rc.order_total) AS lifetime_value,
  COUNT(*) AS total_orders,
  ROUND(AVG(rc.order_total), 2) AS avg_order_value,
  CASE
    WHEN SUM(rc.order_total) > 50000 THEN 'Platinum'
    WHEN SUM(rc.order_total) > 10000 THEN 'Gold'
    WHEN SUM(rc.order_total) > 1000 THEN 'Silver'
    ELSE 'Bronze'
  END AS value_tier
FROM "rest-catalog".ecommerce.orders rc
JOIN "postgres-crm".public.customers pg ON rc.customer_id = pg.customer_id
GROUP BY rc.customer_id, pg.customer_name, pg.region;

In the Catalog, click Edit → Generate Wiki and Generate Tags.

AI-Powered Analytics

Dremio AI Agent

Ask "Who are our Platinum customers?" and the AI Agent generates SQL from your semantic layer. The wiki descriptions you attached explain what "Platinum" means (lifetime value > $50,000), so the Agent produces accurate results.

Dremio MCP Server

Connect Claude or ChatGPT to your catalog data:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs for your AI client
Connect via mcp.dremio.cloud/mcp/{project_id}

A sales director asks Claude "Show me our top 20 Gold and Platinum customers by lifetime value" and gets governed results from your Iceberg catalog.

AI SQL Functions

-- Generate personalized engagement plans
SELECT
  customer_name,
  value_tier,
  lifetime_value,
  AI_GENERATE(
    'Write a one-sentence personalized engagement recommendation',
    'Customer: ' || customer_name || ', Tier: ' || value_tier || ', LTV: $' || CAST(lifetime_value AS VARCHAR) || ', Orders: ' || CAST(total_orders AS VARCHAR && ', Region: ' || region)
  ) AS engagement_plan
FROM analytics.gold.customer_value
WHERE value_tier IN ('Platinum', 'Gold');

-- Classify churn risk
SELECT
  customer_name,
  AI_CLASSIFY(
    'Based on order patterns, classify churn risk',
    'Orders: ' || CAST(total_orders AS VARCHAR) || ', Avg Order: $' || CAST(avg_order_value AS VARCHAR) || ', LTV: $' || CAST(lifetime_value AS VARCHAR),
    ARRAY['Low Risk', 'Moderate Risk', 'High Risk']
  ) AS churn_risk
FROM analytics.gold.customer_value;

Reflections for Performance

Create Reflections on views to cache results and serve BI dashboards with sub-second response times.

When to Use REST Catalog vs. Other Iceberg Catalogs

Use REST Catalog when: Your organization uses Tabular, Apache Polaris, Gravitino, or another REST-compliant catalog server; you need a vendor-neutral catalog interface; you want portability across different compute engines (Dremio, Spark, Trino, Flink).

Use AWS Glue when: You're primarily in the AWS ecosystem and want tight integration with EMR, Athena, and AWS-native tools.

Use Dremio's Open Catalog when: You want zero-configuration automatic table maintenance, Autonomous Reflections, and no external catalog setup.

You can use multiple catalogs simultaneously — for example, REST Catalog for cross-engine shared tables and Dremio's Open Catalog for Dremio-specific analytical workloads.

Governance on REST Catalog Data

Dremio's Fine-Grained Access Control (FGAC) adds governance that REST catalogs don't provide:

Column masking: Mask sensitive columns from specific roles. A business analyst sees aggregated metrics but not individual customer data.
Row-level filtering: Restrict data by the querying user's role. Regional users see only their data.
Unified policies: Same governance applies across REST Catalog, database sources, and other catalogs.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC for BI tools:

Tableau: Dremio connector for direct Arrow Flight access
Power BI: Dremio ODBC driver or native connector
Python/Pandas: pyarrow.flight for programmatic data access
Looker: Connect via JDBC
dbt: dbt-dremio adapter for transformation workflows

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query REST Catalog data from their IDE. Ask Copilot "Show me transaction trends from the Iceberg catalog" and get SQL generated using your semantic layer.

REST Catalog Protocol Details

The Iceberg REST Catalog protocol is an HTTP-based interface defined by the Apache Iceberg project. Any catalog that implements this protocol works with Dremio's connector. This includes:

Apache Polaris (Incubating): Open-source REST catalog by Snowflake
Tabular: Managed Iceberg catalog service (now part of Databricks)
Gravitino: Apache-incubating multi-catalog governance platform
Amazon S3 Tables: AWS-managed Iceberg tables with REST API access
Custom implementations: Any service implementing the Iceberg REST spec

Dremio handles authentication through OAuth2 bearer tokens or custom headers, making it compatible with most enterprise authentication systems.

REST Catalog Endpoints

The Iceberg REST specification defines standard endpoints for catalog operations:

Operation	Endpoint	Dremio Support
List namespaces	`GET /v1/namespaces`	✅
List tables	`GET /v1/namespaces/{ns}/tables`	✅
Load table	`GET /v1/namespaces/{ns}/tables/{table}`	✅
Create table	`POST /v1/namespaces/{ns}/tables`	✅
Update table	`POST /v1/namespaces/{ns}/tables/{table}`	✅
Drop table	`DELETE /v1/namespaces/{ns}/tables/{table}`	✅
Get config	`GET /v1/config`	✅

Dremio uses these endpoints to discover tables, read metadata, perform DML operations, and manage table lifecycle — all through standard HTTP.

Multi-Catalog Architecture

Many organizations run multiple Iceberg catalogs for different purposes:

REST Catalog A (Polaris): Shared enterprise data, governed access for all teams
REST Catalog B (S3 Tables): AWS-native data, auto-managed by AWS
Dremio Open Catalog: Dremio-specific analytical workloads with Autonomous Reflections
AWS Glue: Legacy Iceberg tables managed by existing EMR pipelines

Dremio connects to all of them simultaneously and federates across them. Views in the semantic layer can join tables from different catalogs:

CREATE VIEW analytics.gold.unified_orders AS
SELECT o.order_id, o.order_total, c.customer_name, i.inventory_status
FROM "polaris-catalog".ecommerce.orders o
JOIN "s3-tables".customers.profiles c ON o.customer_id = c.customer_id
JOIN "glue-catalog".warehouse.inventory i ON o.product_id = i.product_id;

Get Started

Iceberg REST Catalog users can query, write, federate, and AI-enrich their Iceberg tables through Dremio Cloud — with governance, Reflections, and AI capabilities that no compute engine provides natively. The REST Catalog connector is the most future-proof choice for organizations adopting Iceberg: as new catalog implementations emerge (and the Iceberg ecosystem is expanding rapidly), this single connector supports them all.

Start by connecting your REST catalog to Dremio Cloud, building a semantic layer over your most important tables, and enabling the AI Agent for natural language querying. The same views and Reflections work regardless of which REST catalog implementation you use — Apache Polaris today, S3 Tables tomorrow, or a custom catalog next year.

Try Dremio Cloud free for 30 days and connect your REST catalog.

Connect Databricks Unity Catalog to Dremio Cloud: Query Delta Lake Tables with Federation and AI

Alex Merced — Mon, 02 Mar 2026 02:00:00 GMT

Databricks Unity Catalog is Databricks' governance layer for data and AI assets. It manages Delta Lake tables, machine learning models, feature stores, and other data objects across Databricks workspaces. If your data engineering team uses Databricks for ETL and ML, your curated analytical datasets likely live in Unity Catalog as Delta Lake tables.

With UniForm, Databricks generates Iceberg-compatible metadata for Delta Lake tables, making them readable by non-Databricks engines without data conversion. This is where Dremio Cloud enters the picture: connect to Unity Catalog through the UniForm Iceberg compatibility layer and query your Delta Lake tables alongside every other data source in your organization — with federation, governance, AI analytics, and performance acceleration that Databricks alone doesn't provide.

Why Unity Catalog Users Need Dremio

Multi-Engine Analytics Beyond Databricks

Unity Catalog centralizes governance for Databricks. But your data consumers use tools beyond Databricks notebooks — Tableau, Power BI, custom Python applications, and business analysts who work in SQL. Dremio provides a high-performance SQL layer that serves all these tools via Arrow Flight (10-100x faster than JDBC/ODBC) or standard ODBC connections.

Instead of provisioning Databricks SQL warehouses for BI workloads (which consume Databricks Units), route those queries through Dremio where Reflections cache results and Autonomous Reflections automatically optimize query performance.

Federate with Non-Databricks Sources

Your Delta Lake tables in Unity Catalog contain curated, processed analytics data. But your operational databases (PostgreSQL, SQL Server, Oracle) live outside Databricks. Your cloud warehouses (Snowflake, Redshift) hold other analytical datasets. Your raw files (S3, Azure Storage) contain event logs and unstructured data. Without a federation layer, combining these with Delta Lake data requires Databricks ingestion pipelines for each source.

Dremio queries each source in place and joins them in a single SQL statement — no ingestion required.

Unified Governance Beyond Databricks

Unity Catalog governs data within Databricks. Dremio's Fine-Grained Access Control (FGAC) governs data across Unity Catalog, PostgreSQL, S3, BigQuery, and every other connected source. One set of column masking and row-level filtering policies, applied consistently everywhere.

AI Analytics on Delta Lake Data

Dremio's semantic layer, AI Agent, MCP Server, and AI SQL Functions add capabilities that Databricks' Genie doesn't replicate — particularly for cross-source analytics and integration with external AI clients like Claude and ChatGPT.

Credential Vending

Unity Catalog supports credential vending across AWS, Azure, and GCS. This means Dremio doesn't need separate S3 or Azure Storage credentials to access the underlying data files — the catalog provides temporary, scoped credentials automatically.

Prerequisites

Databricks workspace URL — your Databricks deployment URL (e.g., https://mycompany.cloud.databricks.com)
Personal Access Token (PAT) or OAuth credentials for Databricks
UniForm enabled on the Delta Lake tables you want to query (this generates Iceberg-compatible metadata)
Storage configuration — AWS, Azure, or GCS (credential vending handles this if configured)
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Enabling UniForm on Delta Lake Tables

To make Delta Lake tables readable from Dremio, enable UniForm in Databricks:

-- In Databricks, enable UniForm when creating a table
CREATE TABLE my_catalog.my_schema.my_table (
  id BIGINT,
  name STRING,
  value DOUBLE
) TBLPROPERTIES (
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

-- Or alter an existing table
ALTER TABLE my_catalog.my_schema.my_table SET TBLPROPERTIES (
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Step-by-Step: Connect Unity Catalog to Dremio Cloud

1. Add the Source

In the Dremio console, click "+" and select Unity Catalog.

2. Configure Connection Details

Name: A descriptive identifier (e.g., unity-catalog or databricks-lakehouse).
Workspace URL: Your Databricks workspace URL.
Credentials: Personal Access Token or OAuth credentials.

3. Select Catalogs and Schemas

Choose which Unity Catalog catalogs and schemas to expose in Dremio. Only tables with UniForm enabled will be readable.

4. Configure Advanced Settings

Set Reflection Refresh and Metadata schedules. More frequent metadata refreshes help Dremio discover new tables and schema changes faster.

5. Set Privileges and Save

Optionally restrict access, then click Save.

Query Delta Lake Tables via UniForm

From Dremio's perspective, UniForm tables appear as standard Iceberg tables:

-- Query ML model predictions
SELECT
  customer_id,
  churn_probability,
  predicted_ltv,
  prediction_date
FROM "unity-catalog".ml_models.customer_predictions
WHERE churn_probability > 0.7 AND prediction_date >= '2024-06-01'
ORDER BY churn_probability DESC;

Federate with Non-Databricks Sources

Join Delta Lake model outputs with operational data from other systems:

-- Combine ML predictions with CRM data and support logs
SELECT
  uc.customer_id,
  uc.churn_probability,
  uc.predicted_ltv,
  pg.customer_name,
  pg.contract_end_date,
  pg.account_manager,
  s3.last_login_date,
  s3.support_tickets_30d,
  CASE
    WHEN uc.churn_probability > 0.8 AND pg.contract_end_date < CURRENT_DATE + INTERVAL '90' DAY THEN 'Critical - Immediate Action'
    WHEN uc.churn_probability > 0.7 THEN 'High Risk - Outreach Needed'
    WHEN uc.churn_probability > 0.5 THEN 'Watch List'
    ELSE 'Healthy'
  END AS action_required
FROM "unity-catalog".ml_models.customer_predictions uc
JOIN "postgres-crm".public.customers pg ON uc.customer_id = pg.customer_id
LEFT JOIN "s3-logs".activity.user_activity s3 ON uc.customer_id = s3.user_id
WHERE uc.churn_probability > 0.5
ORDER BY uc.churn_probability DESC;

Three data systems (Databricks, PostgreSQL, S3), one query, and actionable churn intervention recommendations.

Build a Semantic Layer

CREATE VIEW analytics.gold.customer_risk_dashboard AS
SELECT
  uc.customer_id,
  pg.customer_name,
  pg.region,
  uc.churn_probability,
  uc.predicted_ltv,
  CASE
    WHEN uc.predicted_ltv > 100000 THEN 'Enterprise'
    WHEN uc.predicted_ltv > 25000 THEN 'Mid-Market'
    ELSE 'SMB'
  END AS value_segment,
  CASE
    WHEN uc.churn_probability > 0.7 THEN 'High Risk'
    WHEN uc.churn_probability > 0.4 THEN 'Moderate Risk'
    ELSE 'Low Risk'
  END AS risk_tier
FROM "unity-catalog".ml_models.customer_predictions uc
JOIN "postgres-crm".public.customers pg ON uc.customer_id = pg.customer_id;

Navigate to the Catalog, click Edit (pencil icon), and Generate Wiki and Generate Tags. This creates business descriptions like "customer_risk_dashboard: Contains one row per customer combining ML churn predictions from Databricks with CRM account details" — context that powers AI features.

AI-Powered Analytics on Delta Lake Data

Dremio AI Agent

The AI Agent lets business users ask questions in plain English: "Which enterprise customers are at high risk of churning?" or "Show me our top 10 customers by predicted lifetime value." The Agent reads your wiki descriptions to understand what "enterprise," "high risk," and "lifetime value" mean in your data context, then generates accurate SQL.

This is particularly powerful for Delta Lake data because model outputs (churn scores, predictions) often need business interpretation. The semantic layer bridges the gap between ML model outputs and business-friendly analytics.

Dremio MCP Server

The Dremio MCP Server connects Claude, ChatGPT, and other AI clients to your Dremio data. Setup:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs (e.g., https://claude.ai/api/mcp/auth_callback for Claude, https://chatgpt.com/connector_platform_oauth_redirect for ChatGPT)
Connect using mcp.dremio.cloud/mcp/{project_id} (US) or mcp.eu.dremio.cloud/mcp/{project_id} (EU)

A customer success manager can ask Claude "Show me all high-risk enterprise customers with contracts ending in the next 90 days" and get accurate, governed results from your Unity Catalog ML predictions — without knowing SQL.

AI SQL Functions

Use AI directly in queries against Unity Catalog data:

-- Generate personalized retention messages for at-risk customers
SELECT
  customer_name,
  churn_probability,
  predicted_ltv,
  AI_GENERATE(
    'Write a one-sentence personalized retention offer for this at-risk customer',
    'Customer: ' || customer_name || ', Segment: ' || value_segment || ', Risk: ' || risk_tier || ', LTV: $' || CAST(predicted_ltv AS VARCHAR)
  ) AS retention_message
FROM analytics.gold.customer_risk_dashboard
WHERE risk_tier = 'High Risk' AND value_segment = 'Enterprise';

-- Classify intervention urgency
SELECT
  customer_name,
  AI_CLASSIFY(
    'Based on these risk factors, classify the intervention urgency',
    'Churn probability: ' || CAST(churn_probability AS VARCHAR) || ', LTV: $' || CAST(predicted_ltv AS VARCHAR) || ', Segment: ' || value_segment,
    ARRAY['Immediate', 'This Week', 'This Month', 'Monitor']
  ) AS urgency
FROM analytics.gold.customer_risk_dashboard
WHERE risk_tier IN ('High Risk', 'Moderate Risk');

Important Notes

Read-only access. Dremio connects to Unity Catalog tables in read-only mode. Write operations continue through Databricks.
UniForm required. Only Delta Lake tables with UniForm enabled appear as queryable Iceberg tables in Dremio.
Table format transparency. From Dremio's perspective, UniForm tables look and behave like standard Iceberg tables.
Credential vending. When configured, Dremio receives temporary credentials from Unity Catalog, simplifying storage access.

Reflections for Performance

Create Reflections on Unity Catalog views to cache results and serve dashboard queries without re-reading Delta Lake files:

Navigate to the view in the Catalog
Click the Reflections tab
Choose Raw Reflection or Aggregation Reflection
Select columns and aggregations
Set the Refresh Interval — balance between data freshness and compute cost
Click Save

BI tools connected to Dremio get sub-second response times from Reflections. This eliminates the need for Databricks SQL warehouses for read-heavy BI workloads.

Governance Across Unity Catalog and Other Sources

Unity Catalog governs data within Databricks. Dremio's Fine-Grained Access Control (FGAC) extends governance across Unity Catalog and every other connected source:

Column masking: Mask churn probability or predicted LTV from specific roles. A sales rep sees risk tier but not raw probability scores.
Row-level filtering: Regional account managers see only customers in their territory.
Unified policies: Same rules apply across Unity Catalog, PostgreSQL, S3, and all other sources.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC. For Delta Lake data:

Tableau: Dremio connector — avoids Databricks SQL warehouse costs for BI
Power BI: Dremio ODBC or native connector
Python/Pandas: pyarrow.flight for programmatic access to ML model outputs
Looker: Connect via JDBC
dbt: dbt-dremio for transformations

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot lets developers query Unity Catalog data from their IDE. Ask Copilot "Show me high-risk enterprise customers from the churn model" and get SQL from your semantic layer — without switching to Databricks notebooks or SQL warehouses.

When to Use Dremio vs. Databricks SQL

Use Dremio when: You need cross-source federation (joining Delta Lake with PostgreSQL, S3, Snowflake), you want AI analytics on federated data, you need governance across multiple data sources, you want Reflection-based caching for BI tools.

Use Databricks SQL when: You need write-heavy workloads on Delta Lake, you're running Databricks-native jobs (streaming, ML training), your queries use Databricks-specific SQL extensions.

Both can coexist — Databricks for data engineering and ML, Dremio for federated analytics, AI, and BI serving.

Delta Lake Tables in Dremio

Dremio reads Delta Lake tables from Unity Catalog with full Delta protocol support:

Time travel: Query tables at specific versions using Delta Lake's transaction log
Schema evolution: Dremio automatically detects schema changes made by Databricks jobs
Partition pruning: Dremio leverages Delta Lake's partition statistics to skip irrelevant data files
Column statistics: Delta Lake's min/max statistics enable efficient predicate pushdown

For write operations, use Dremio's Open Catalog with Iceberg tables for new analytical workloads. Unity Catalog remains the source of truth for Databricks-managed Delta Lake tables.

Databricks Cost Optimization

Databricks pricing is based on Databricks Units (DBUs) consumed by SQL warehouses, clusters, and jobs. Dremio helps optimize costs:

BI serving: Instead of running a Databricks SQL warehouse 24/7 for dashboards, create Reflections in Dremio. Dashboard queries hit Dremio, SQL warehouse auto-stops.
Ad-hoc exploration: Analysts query Dremio's cached Reflections instead of waking Databricks clusters. Less start/stop overhead.
Cross-source queries: Joining Delta Lake with PostgreSQL or S3 doesn't require moving all data into Databricks — Dremio federates in place.

For organizations spending $50K+/month on Databricks, routing read-heavy analytical workloads through Dremio can reduce DBU consumption by 30-50% on those workloads.

Get Started

Unity Catalog users can extend their Databricks investment with Dremio's federation, AI analytics, and performance acceleration — without moving data out of Delta Lake. Dremio and Databricks are complementary: Databricks handles data engineering, ML training, and streaming workloads on Delta Lake tables, while Dremio serves analytical queries, BI dashboards, and AI-powered natural language access across your entire data estate.

Connect your Unity Catalog to Dremio Cloud, build Reflections on frequently queried tables, and enable the AI Agent for business users who need answers without writing SQL.

Try Dremio Cloud free for 30 days and connect your Unity Catalog.

Connect Snowflake Open Catalog to Dremio Cloud: Multi-Engine Iceberg Analytics

Alex Merced — Mon, 02 Mar 2026 01:00:00 GMT

Snowflake Open Catalog is Snowflake's managed implementation of the Apache Iceberg REST catalog specification, based on the open-source Apache Polaris project. It serves as a centralized metadata catalog for Apache Iceberg tables, enabling multiple compute engines — including Dremio, Spark, Trino, and Flink — to read from and write to the same Iceberg tables without metadata conflicts.

Dremio Cloud connects to Snowflake Open Catalog as a first-class Iceberg data source. You get full read and write access to Iceberg tables, automatic table maintenance (compaction, manifest optimization, vacuuming), and the ability to federate catalog data with databases, object storage, cloud warehouses, and other catalogs — all through standard SQL.

For organizations already invested in Snowflake, the Open Catalog is a strategic choice for multi-engine interoperability. Unlike Snowflake's proprietary internal catalog (which is only accessible through Snowflake compute), the Open Catalog exposes Iceberg metadata via a standard REST API. This means you're not locked into Snowflake compute for every analytical query — Dremio can read the same tables at a fraction of the credit cost for repetitive workloads. Dremio also provides its federated engine, Reflections, governance, and AI capabilities — all without duplicating data or metadata.

Why Snowflake Open Catalog Users Need Dremio

Multi-Engine Strategy Without Vendor Lock-In

Snowflake Open Catalog is designed for multi-engine compatibility, which makes it an ideal complement to Dremio. By connecting Dremio to your Snowflake Open Catalog, you add a query engine that specializes in areas Snowflake doesn't:

Federation: Join catalog tables with PostgreSQL, MongoDB, S3, BigQuery, and any other Dremio-connected source in a single SQL query — something Snowflake can't do natively with non-Snowflake sources.
Autonomous performance management: Dremio automatically compacts files, rewrites manifests, and builds Reflections based on query patterns for external catalog tables.
AI-powered querying: Dremio's AI Agent, MCP Server, and AI SQL Functions bring LLM capabilities to your catalog data.

Cost Optimization

Instead of running all workloads through Snowflake credits, offload analytical queries to Dremio. Dremio's Reflections cache results so repeated queries don't consume Snowflake credits. For organizations spending significant amounts on Snowflake compute, routing read-heavy analytical workloads through Dremio can reduce overall costs.

Federate with Non-Snowflake Sources

Snowflake's data sharing works within Snowflake. But what if you need to join your Snowflake Open Catalog data with PostgreSQL application data, MongoDB user profiles, or S3 raw event logs? Dremio's federation engine does exactly that — no ETL pipelines, no data duplication.

Credential Vending

Snowflake Open Catalog supports credential vending, meaning Dremio doesn't need separate storage credentials to access the underlying S3, Azure, or GCS data. The catalog provides temporary, scoped credentials for accessing data files. This simplifies security configuration and reduces the credentials you need to manage.

Write Support for External Catalogs

Dremio can write to external Snowflake Open Catalogs, enabling you to create tables, run transformations, and build data pipelines using Dremio's SQL engine while keeping metadata managed in Snowflake's catalog.

Prerequisites

Before connecting to Snowflake Open Catalog, confirm you have:

Snowflake Open Catalog account URL — the endpoint for your catalog instance
OAuth or Personal Access Token (PAT) credentials — for authenticating to the catalog
Catalog names — the specific catalogs you want to access (internal read-only and/or external read-write)
Storage configuration — if credential vending isn't available for your setup, you'll need S3, Azure, or GCS credentials for the underlying data
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Snowflake Open Catalog to Dremio Cloud

1. Add the Source

In the Dremio console, click the "+" button in the left sidebar and select Snowflake Open Catalog from the catalog source types.

2. Configure Connection Details

Name: A descriptive identifier (e.g., snowflake-open-catalog or lakehouse-catalog). This appears in SQL queries as the source prefix.
Catalog URL: The Snowflake Open Catalog endpoint.
Credentials: OAuth client ID/secret or a Personal Access Token.

3. Select Catalogs

Choose which catalogs to enable:

Internal catalogs are read-only from Dremio's perspective — you can query but not write.
External catalogs support full read and write operations (INSERT, UPDATE, DELETE, MERGE).

4. Configure Advanced Settings

Set Reflection Refresh and Metadata schedules. For catalogs with frequently changing tables, more frequent metadata refreshes ensure Dremio sees new tables and schema changes quickly.

5. Set Privileges and Save

Optionally restrict which Dremio users can access this catalog. Click Save.

Query Snowflake Open Catalog Data

-- Query an Iceberg table managed by Snowflake Open Catalog
SELECT customer_id, customer_name, total_spend, signup_date
FROM "sf-open-catalog".analytics.customer_summary
WHERE total_spend > 10000 AND signup_date >= '2024-01-01'
ORDER BY total_spend DESC;

-- Write to an external catalog
INSERT INTO "sf-open-catalog".analytics.monthly_metrics
SELECT
  DATE_TRUNC('month', order_date) AS month,
  COUNT(*) AS order_count,
  SUM(total_amount) AS revenue
FROM "sf-open-catalog".ecommerce.orders
GROUP BY 1;

Federate with Other Sources

Join catalog data with non-Snowflake sources in a single query:

SELECT
  soc.customer_name,
  soc.total_spend AS catalog_spend,
  pg.region,
  pg.account_manager,
  s3.support_ticket_count,
  CASE
    WHEN soc.total_spend > 100000 AND s3.support_ticket_count < 3 THEN 'Platinum'
    WHEN soc.total_spend > 50000 THEN 'Gold'
    WHEN soc.total_spend > 10000 THEN 'Silver'
    ELSE 'Standard'
  END AS customer_tier
FROM "sf-open-catalog".analytics.customer_summary soc
JOIN "postgres-crm".public.customers pg ON soc.customer_id = pg.customer_id
LEFT JOIN "s3-support".tickets.customer_tickets s3 ON soc.customer_id = s3.customer_id
ORDER BY catalog_spend DESC;

Build a Semantic Layer

Create views that combine catalog data with business logic:

CREATE VIEW analytics.gold.customer_health AS
SELECT
  soc.customer_id,
  soc.customer_name,
  soc.total_spend,
  soc.signup_date,
  CASE
    WHEN soc.total_spend > 100000 THEN 'Enterprise'
    WHEN soc.total_spend > 25000 THEN 'Mid-Market'
    ELSE 'SMB'
  END AS customer_segment,
  ROUND(soc.total_spend / GREATEST(DATEDIFF('MONTH', soc.signup_date, CURRENT_DATE), 1), 2) AS monthly_spend_rate
FROM "sf-open-catalog".analytics.customer_summary soc;

Navigate to the Catalog, click Edit (pencil icon) on this view, and Generate Wiki and Generate Tags. This creates the business context that powers Dremio's AI features.

AI-Powered Analytics

Dremio AI Agent

The AI Agent lets users ask questions in plain English. For example: "Who are our highest-spending enterprise customers?" The Agent reads your wiki descriptions and view definitions to generate the correct SQL. Better wikis produce better results — describe what "enterprise customer" and "monthly spend rate" mean in business terms.

Dremio MCP Server

The Dremio MCP Server extends AI capabilities to Claude, ChatGPT, and other AI chat clients. Connect through the hosted MCP Server:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs for your AI client (e.g., https://claude.ai/api/mcp/auth_callback)
Connect using mcp.dremio.cloud/mcp/{project_id}

Your team can then ask Claude "Show me customer health trends from our Snowflake catalog data" and get governed, accurate results without writing SQL.

AI SQL Functions

Enrich catalog data with AI inline in your queries:

SELECT
  customer_name,
  total_spend,
  AI_CLASSIFY(
    'Based on spending patterns, classify customer risk of churn',
    'Customer: ' || customer_name || ', Total Spend: $' || CAST(total_spend AS VARCHAR) || ', Months Active: ' || CAST(months_active AS VARCHAR),
    ARRAY['Low Risk', 'Moderate Risk', 'High Risk', 'Critical']
  ) AS churn_risk
FROM "sf-open-catalog".analytics.customer_summary
WHERE total_spend > 5000;

AI_CLASSIFY runs LLM inference in your SQL query. AI_GENERATE produces narrative summaries, and AI_SIMILARITY finds semantic matches between text fields.

Reflections for Performance

Create Reflections on frequently queried views to cache results:

In the Catalog, select the view and click the Reflections tab
Choose Raw Reflection (full cache) or Aggregation Reflection (pre-computed metrics)
Select columns and aggregations to include
Set the Refresh Interval — balance freshness against compute cost
Click Save

BI tools connected via Arrow Flight or ODBC get sub-second responses from Reflections instead of re-reading Iceberg files from storage. This reduces Snowflake credit consumption for workloads routed through Dremio.

Governance Across Snowflake Open Catalog and Other Sources

Dremio's Fine-Grained Access Control (FGAC) adds governance that spans Snowflake Open Catalog and all other sources:

Column masking: Mask sensitive customer data from specific roles. A marketing analyst sees spending behavior but not PII.
Row-level filtering: Regional users see only their region's data automatically.
Unified policies: One set of governance rules applies across Snowflake Open Catalog, database connectors, and other external catalogs.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Tableau: Dremio connector for direct Arrow Flight access
Power BI: Dremio ODBC driver or native connector
Python/Pandas: pyarrow.flight for high-speed programmatic access
Looker: Connect via JDBC
dbt: dbt-dremio adapter for transformation workflows

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Snowflake Open Catalog data from their IDE. Ask Copilot "Show me customer churn risk from the catalog" and get SQL generated using your semantic layer — without switching tools.

When to Use Snowflake Open Catalog vs. Other Catalogs

Use Snowflake Open Catalog when: You're already in the Snowflake ecosystem and want multi-engine Iceberg access, your team uses Snowflake for data management but needs Dremio for federation and AI.

Use AWS Glue when: You're AWS-native and want tight integration with EMR, Athena, and S3.

Use Dremio's Open Catalog when: You want zero-configuration automatic maintenance, Autonomous Reflections, and no external catalog dependencies.

You can connect multiple catalogs simultaneously. Many organizations use Snowflake Open Catalog for shared enterprise data and Dremio's Open Catalog for Dremio-specific analytical workloads.

Credential Vending in Detail

Credential vending is a key feature of Snowflake Open Catalog that simplifies Dremio's access to underlying storage. Here's how it works:

You configure storage in Snowflake Open Catalog — specify the S3, Azure, or GCS bucket where Iceberg data files live.
When Dremio queries a table, it requests access from the catalog API.
Snowflake Open Catalog returns temporary, scoped credentials — short-lived tokens with permissions limited to the specific data files needed.
Dremio uses these credentials to read (or write, for external catalogs) directly from storage.
Credentials expire automatically — no long-lived keys to rotate or manage.

This means your Dremio Cloud connection needs only the catalog API credentials (OAuth), not separate storage credentials for every S3 bucket or Azure container. One connection, automatic credential management, reduced security surface area.

Multi-Engine Architecture with Snowflake Open Catalog

Snowflake Open Catalog enables a powerful multi-engine architecture:

Snowflake: Data engineering, SQL analytics, and catalog management
Dremio: Federation, AI analytics, and Reflection-based BI serving
Apache Spark: Large-scale data processing and ML model training
Trino/Presto: Ad-hoc query engine for open-source workflows

All engines read from the same Iceberg tables managed by Snowflake Open Catalog — no data duplication, no metadata sync issues, no format conversion. Each engine reads the latest table metadata from the catalog and accesses data files via credential vending.

Dremio's unique contribution to this architecture is federation (joining catalog tables with non-Iceberg sources), AI capabilities (Agent, MCP, SQL Functions), and Reflections (sub-second BI serving without re-reading storage).

Snowflake Open Catalog vs. Apache Polaris

Snowflake Open Catalog is based on the open-source Apache Polaris (incubating) project. Key differences:

Feature	Snowflake Open Catalog	Apache Polaris (self-managed)
Hosting	Managed by Snowflake	Self-hosted
Credential Vending	Built-in	Requires configuration
Authentication	Snowflake OAuth	Custom
Support	Snowflake support	Community
Cost	Snowflake pricing	Infrastructure costs

If you use Snowflake's managed offering, you get turnkey catalog management. If you prefer self-managed, Apache Polaris works with Dremio's Iceberg REST Catalog connector.

Get Started

Snowflake Open Catalog users can build a truly multi-engine lakehouse — manage Iceberg metadata in Snowflake's infrastructure while querying with Dremio's federated engine, AI capabilities, and Reflection-based acceleration.

Connect your Snowflake Open Catalog to Dremio Cloud, build views over your Iceberg tables, and start leveraging AI Agent, MCP Server, and Reflections for cost-optimized analytical serving. The setup takes minutes and works immediately.

Try Dremio Cloud free for 30 days and connect your Snowflake Open Catalog.

Connect AWS Glue Data Catalog to Dremio Cloud: Query and Manage Your AWS Iceberg Tables

Alex Merced — Mon, 02 Mar 2026 00:00:00 GMT

AWS Glue Data Catalog is AWS's managed metadata service for data lakes. It stores table definitions, schemas, partition information, and statistics for data stored in Amazon S3. If you've built your data lake on AWS using Apache Spark (on EMR), AWS Glue ETL jobs, or Amazon Athena, your table metadata lives in Glue. But Glue is just a catalog — a registry of what's where. To actually query the data, you need Athena (per-TB pricing), EMR clusters (infrastructure management), or Redshift Spectrum (additional cost).

Dremio Cloud connects to your Glue Data Catalog and queries the underlying Iceberg tables with full read and write support. You get enterprise-grade SQL, Reflections for query acceleration, governance, and AI analytics — all on top of your existing Glue-managed lakehouse.

Why Glue Users Need Dremio

Query Without Athena's Per-TB Pricing

Athena charges per terabyte of data scanned, regardless of whether the query is the same one you ran 5 minutes ago. For teams running dashboard queries, scheduled reports, and ad-hoc exploration, this pricing model creates unpredictable costs. Dremio's Reflections cache results so repeated queries don't re-scan S3. C3 (Columnar Cloud Cache) caches file data on local NVMe for frequently accessed datasets. You pay for Dremio compute time, not per-TB scanned.

Full Read and Write on Iceberg Tables

Dremio supports full DML (INSERT, UPDATE, DELETE, MERGE) on Glue-cataloged Iceberg tables. Create tables, run transformations, build data pipelines, and maintain your lakehouse entirely through Dremio's SQL engine — no need to spin up EMR clusters or Glue ETL jobs for simple transformations.

Federate Glue with Non-AWS Sources

Your Glue-managed data lake covers AWS data, but your application database is on Azure (Azure SQL), your analytics warehouse is Snowflake, and your marketing data is in Google BigQuery. Dremio federates across all of them in a single SQL query.

Automated Iceberg Maintenance

Dremio automatically compacts small files into optimally sized ones, rewrites manifests for faster metadata reads, and clusters data based on query patterns — all on Glue-cataloged Iceberg tables. This eliminates the need for manual OPTIMIZE jobs or scheduled Glue ETL maintenance tasks.

Credential Vending

Dremio uses Glue's credential vending to securely access the underlying S3 data without separate S3 credentials. The catalog provides temporary, scoped credentials for each data request.

Prerequisites

AWS Account with Glue Data Catalog configured
IAM Role with permissions: glue:GetDatabase, glue:GetTable, glue:GetTables, glue:GetPartitions, and S3 read/write permissions for underlying data
AWS Region where your Glue catalog is deployed
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Glue to Dremio Cloud

1. Add the Source

Click "+" in the Dremio console and select AWS Glue Data Catalog.

2. Configure Connection

Name: Descriptive identifier (e.g., glue-catalog or aws-lakehouse).
AWS Region: The region where your Glue catalog is deployed.

3. Set Authentication

Provide IAM Role ARN (recommended for Dremio Cloud) or AWS Access Key/Secret Key.

4. Select Databases

Choose which Glue databases to expose. You can enable specific databases or allow access to all.

5. Configure Advanced Settings

Set Reflection Refresh, Metadata refresh intervals, and click Save.

Query and Write to Glue Iceberg Tables

-- Query a Glue-cataloged Iceberg table
SELECT product_id, product_name, category, price, inventory_count
FROM "glue-catalog".ecommerce.products
WHERE category = 'Electronics' AND price > 50 AND inventory_count > 0
ORDER BY price ASC;

-- Write to Glue Iceberg tables
INSERT INTO "glue-catalog".analytics.daily_summary
SELECT
  DATE_TRUNC('day', order_date) AS day,
  COUNT(*) AS order_count,
  SUM(total) AS revenue,
  AVG(total) AS avg_order_value
FROM "glue-catalog".ecommerce.orders
WHERE order_date = CURRENT_DATE - INTERVAL '1' DAY
GROUP BY 1;

-- MERGE for upserts
MERGE INTO "glue-catalog".analytics.product_metrics AS target
USING (
  SELECT product_id, COUNT(*) AS orders, SUM(quantity) AS units_sold
  FROM "glue-catalog".ecommerce.order_items
  WHERE order_date >= CURRENT_DATE - INTERVAL '7' DAY
  GROUP BY product_id
) AS source
ON target.product_id = source.product_id
WHEN MATCHED THEN UPDATE SET orders = source.orders, units_sold = source.units_sold
WHEN NOT MATCHED THEN INSERT (product_id, orders, units_sold) VALUES (source.product_id, source.orders, source.units_sold);

Federate with Non-AWS Sources

-- Join Glue products with external review and supplier data
SELECT
  g.product_name,
  g.price,
  g.category,
  pg.avg_rating,
  pg.review_count,
  sf.supplier_name,
  sf.lead_time_days
FROM "glue-catalog".ecommerce.products g
LEFT JOIN "postgres-reviews".public.product_reviews pg ON g.product_id = pg.product_id
LEFT JOIN "snowflake-supply".PUBLIC.SUPPLIERS sf ON g.supplier_id = sf.supplier_id
WHERE g.category = 'Electronics'
ORDER BY pg.avg_rating DESC;

Build a Semantic Layer

CREATE VIEW analytics.gold.product_performance AS
SELECT
  g.product_id,
  g.product_name,
  g.category,
  g.price,
  SUM(oi.quantity) AS units_sold,
  SUM(oi.quantity * g.price) AS revenue,
  CASE
    WHEN SUM(oi.quantity) > 1000 THEN 'Best Seller'
    WHEN SUM(oi.quantity) > 100 THEN 'Popular'
    ELSE 'Niche'
  END AS popularity_tier
FROM "glue-catalog".ecommerce.products g
LEFT JOIN "glue-catalog".ecommerce.order_items oi ON g.product_id = oi.product_id
GROUP BY g.product_id, g.product_name, g.category, g.price;

In the Catalog, click Edit → Generate Wiki and Generate Tags.

AI-Powered Analytics

Dremio AI Agent

Ask "Which electronics products are best sellers?" and the AI Agent generates SQL from your semantic layer. The wiki descriptions you've attached to views guide the Agent's understanding of terms like "best seller" and "popularity tier."

Dremio MCP Server

The Dremio MCP Server connects Claude and ChatGPT to your Glue-cataloged data:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs for your AI client
Connect using mcp.dremio.cloud/mcp/{project_id}

A product manager asks ChatGPT "Show me niche electronics products with high ratings that might be under-marketed" and gets governed results from your Glue lakehouse.

AI SQL Functions

-- Generate product descriptions from catalog data
SELECT
  product_name,
  category,
  price,
  AI_GENERATE(
    'Write a one-sentence marketing description for this product',
    'Product: ' || product_name || ', Category: ' || category || ', Price: $' || CAST(price AS VARCHAR) || ', Popularity: ' || popularity_tier
  ) AS marketing_description
FROM analytics.gold.product_performance
WHERE popularity_tier = 'Best Seller';

-- Classify inventory risk
SELECT
  product_name,
  inventory_count,
  AI_CLASSIFY(
    'Based on inventory levels and sales velocity, classify the reorder urgency',
    'Product: ' || product_name || ', Stock: ' || CAST(inventory_count AS VARCHAR) || ', Units Sold (7d): ' || CAST(units_sold AS VARCHAR),
    ARRAY['Order Now', 'Order Soon', 'Adequate Stock', 'Overstocked']
  ) AS reorder_urgency
FROM "glue-catalog".ecommerce.products g
JOIN analytics.gold.product_performance pp ON g.product_id = pp.product_id;

Reflections for Performance

Create Reflections on product performance and daily summary views to cache results and serve BI tools with sub-second response times:

In the Catalog, navigate to the view you want to accelerate
Click the Reflections tab
Choose Raw Reflection to cache the full view or Aggregation Reflection to pre-compute specific SUM/COUNT/AVG aggregations
Select columns to include in the Reflection
Set the Refresh Interval — how often Dremio re-queries the underlying Iceberg tables to update the cache
Click Save

Dashboard queries from Tableau, Power BI, or Looker connected via Arrow Flight hit the Reflection instead of re-reading S3 Iceberg files, providing sub-second response times even for complex aggregations.

Time Travel on Glue Iceberg Tables

Iceberg tables cataloged in Glue support time travel through Dremio:

-- Query a table as it existed 7 days ago
SELECT product_id, price, inventory_count
FROM "glue-catalog".ecommerce.products
AT TIMESTAMP '2024-06-01 00:00:00';

-- Compare current state to a historical snapshot
SELECT
  curr.product_name,
  curr.price AS current_price,
  hist.price AS previous_price,
  ROUND((curr.price - hist.price) / hist.price * 100, 2) AS price_change_pct
FROM "glue-catalog".ecommerce.products curr
JOIN "glue-catalog".ecommerce.products AT TIMESTAMP '2024-01-01 00:00:00' hist
  ON curr.product_id = hist.product_id
WHERE curr.price != hist.price
ORDER BY ABS(price_change_pct) DESC;

Time travel is valuable for auditing ("What were inventory levels at quarter end?"), debugging ("What changed in the last 24 hours?"), and compliance ("Show data as it was on the regulatory reporting date").

Governance on Glue Data

Dremio's Fine-Grained Access Control (FGAC) adds governance capabilities that Glue and Athena don't provide natively:

Column masking: Hide sensitive fields (customer PII, pricing details) from specific roles while allowing full access for authorized users. For example, mask customer_email for marketing analysts but show it for customer support teams.
Row-level filtering: Automatically filter data based on the querying user's role. A regional manager sees only their region's data. A global admin sees everything.
Unified policies: The same governance policies apply whether data comes from Glue, PostgreSQL, Snowflake, or any other connected source.

These policies apply across all access methods — SQL Runner, BI tools via Arrow Flight/ODBC, AI Agent queries, and MCP Server interactions.

Connect BI Tools via Arrow Flight

Dremio's Arrow Flight connector provides 10-100x faster data transfer compared to JDBC/ODBC for BI tools. After creating views over your Glue data, connect:

Tableau: Use the Dremio connector, enter your Dremio Cloud endpoint
Power BI: Use the Dremio ODBC driver or Arrow Flight connector
Python/Pandas: Use pyarrow.flight client for high-speed data access
dbt: Use the dbt-dremio adapter for transformation workflows

All queries from these tools benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Glue-cataloged data directly from their IDE. Ask Copilot "Show me product inventory trends from the Glue catalog" and it generates SQL using Dremio's semantic layer — all without leaving your development environment.

Glue vs. Athena vs. Dremio: When to Use Each

Feature	AWS Glue	Amazon Athena	Dremio Cloud
Purpose	Metadata catalog	Serverless SQL	Federated analytics + catalog
Pricing	Free (metadata)	Per TB scanned	Compute-based
Write support	Via ETL jobs	Limited	Full DML
Federation	No	Federated queries (limited)	Full cross-source federation
AI analytics	No	No	AI Agent, MCP, SQL Functions
Reflections	No	No	Yes (automatic caching)
Governance	IAM only	IAM + Lake Formation	FGAC + semantic layer

Glue is the metadata catalog. Athena is a query engine with per-TB pricing. Dremio is a federated platform that uses Glue as one of many catalogs and adds AI, governance, and performance acceleration.

When to Keep Tables in Glue vs. Use Dremio's Open Catalog

Keep in Glue: Tables managed by existing AWS-native pipelines (EMR, Glue ETL), tables shared across multiple AWS services, data consumed by Athena or Redshift Spectrum alongside Dremio.

Use Dremio's Open Catalog: New analytical tables, data created through Dremio transformations, datasets where you want zero-configuration automatic maintenance (compaction, vacuuming, Autonomous Reflections).

You can use both simultaneously — Glue for your existing AWS lakehouse, Dremio's Open Catalog for new analytical workloads.

Dremio vs. Athena for Querying Glue-Managed Tables

Both Dremio and Athena can query tables registered in the Glue Data Catalog. Key differences:

Feature	Dremio Cloud	Amazon Athena
Pricing	Compute-based	$5/TB scanned
Reflections	✅ Cache results	❌ Scans every time
Federation	PostgreSQL, MongoDB, BigQuery, etc.	S3 + federated queries (limited)
AI Agent	✅ Natural language queries	❌
MCP Server	✅ Claude/ChatGPT integration	❌
BI Tool Connectivity	Arrow Flight (10-100x faster)	ODBC/JDBC only
Governance	Column masking + row filtering	Lake Formation policies
Iceberg Write Support	Full DML	Full DML

For organizations already using Athena, Dremio adds federation, AI analytics, and cost savings through Reflections. Many teams run both: Athena for quick ad-hoc S3 queries, Dremio for cross-source analytics and BI tool serving.

AWS Lake Formation Integration

AWS Lake Formation provides fine-grained access control for Glue-managed tables. When connecting to Glue through Dremio:

Lake Formation permissions govern which tables and columns the Dremio IAM role can access
Dremio FGAC adds additional governance layers (column masking, row-level filtering) on top of Lake Formation
Both layers work together: Lake Formation controls what Dremio can see; Dremio FGAC controls what individual users see

This dual-layer governance model gives you AWS-native access control at the storage level and Dremio-managed access control at the query level — comprehensive governance without compromising on either side.

Get Started

AWS Glue Data Catalog users can query, write, optimize, and AI-enrich their Iceberg tables through Dremio Cloud — with federation, governance, and performance acceleration that Athena and EMR don't provide.

Try Dremio Cloud free for 30 days and connect your Glue catalog.

Connect Apache Druid to Dremio Cloud: Add SQL Joins, AI, and Governance to Your Real-Time Analytics

Alex Merced — Sun, 01 Mar 2026 23:00:00 GMT

Apache Druid is a real-time analytics database designed for sub-second queries on high-ingestion-rate event data. Clickstream analytics, application monitoring, IoT telemetry, and ad-tech workloads rely on Druid's columnar storage and inverted indexes for instantaneous queries.

Dremio Cloud connects to Druid as a federated data source, giving you the ability to join Druid event data with relational databases, data lakes, and cloud warehouses. Dremio adds governance (column masking, row-level filtering), Reflection-based acceleration, and AI capabilities (AI Agent, MCP Server, AI SQL Functions) that Druid doesn't provide natively.

Druid excels at one thing: fast aggregation queries on time-series event data. But production analytics rarely involve just one data source. When a product manager asks "Show me user engagement metrics correlated with support ticket volume and revenue impact," that query requires joining Druid's event data with a CRM database and a financial system. Druid can't do these joins natively — it doesn't support standard SQL JOINs. Dremio bridges this gap by reading Druid data and joining it with any other source in a single SQL query.

But Druid has fundamental limitations that become painful as analytics needs grow. It doesn't support traditional SQL joins between datasources. It doesn't connect to external databases. Its query model is optimized for aggregations on its own ingested segments, not for the kind of cross-source, enriched analytics modern organizations need.

Dremio Cloud connects to Apache Druid and queries it alongside relational databases, data lakes, and cloud warehouses. You get the speed of Druid for real-time aggregations combined with Dremio's ability to join that data with any other source, accelerate queries with Reflections, apply governance, and enable AI analytics.

Why Druid Users Need Dremio

SQL Joins with Real-Time Data

Druid doesn't support traditional SQL joins between datasources. If you want to answer "What is the conversion rate by customer segment in the last hour?" you need the real-time event data from Druid and the customer segment data from your CRM database. Without Dremio, you'd need to either pre-join the data before ingesting into Druid (losing flexibility) or build application code that queries both systems and merges results in memory.

Dremio queries Druid for its real-time aggregations and joins the results with PostgreSQL customer data, S3 behavior logs, Snowflake revenue data, or any other connected source — all in a single SQL query.

Enrich Real-Time Metrics with Business Context

Druid provides fast counts, averages, percentiles, and approximate distinct counts on event data. But enriching those metrics with customer names, product descriptions, geographic hierarchies, or organizational data requires joining with dimensional data that lives in other systems.

Dremio's federation provides that enrichment without duplicating dimensional data into Druid. Your Druid segments stay lean (just events), and Dremio handles the enrichment at query time.

Historical Analysis Across Time Ranges

Druid is optimized for recent data (hot segments). Historical analysis across months or years — trend analysis, year-over-year comparisons — often hits cold segments that are slower to query. Dremio's Reflections cache aggregated historical results, providing fast access to time-series trends without depending on Druid's tiered storage.

Unified Governance

Druid has basic authentication but limited access control. There's no column masking, no row-level filtering, no consistent policy framework. Dremio's Fine-Grained Access Control adds these capabilities, ensuring that sensitive event data (user IDs, IP addresses, location data) is properly governed across Druid and every other connected source.

Prerequisites

Druid Broker hostname or IP address — the Broker node handles query routing
Port — typically 8082 for the Broker HTTP API
Network access from Dremio Cloud to the Druid Broker
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Druid to Dremio Cloud

1. Add the Source

In the Dremio console, click "+" and select Apache Druid.

2. Configure Connection Details

Name: A descriptive identifier (e.g., druid-realtime or event-analytics).
Host: Druid Broker hostname or IP.
Port: Default 8082.

3. Set Authentication

Configure credentials if your Druid deployment requires authentication.

4. Configure Advanced Settings

Set Reflection Refresh, Metadata refresh intervals, and any connection properties.

5. Set Privileges and Save

Query Real-Time Druid Data

-- Real-time page view metrics
SELECT
  DATE_TRUNC('hour', __time) AS event_hour,
  page,
  COUNT(*) AS page_views,
  COUNT(DISTINCT user_id) AS unique_visitors,
  ROUND(COUNT(*) * 1.0 / COUNT(DISTINCT user_id), 2) AS views_per_visitor
FROM "druid-realtime".druid.pageviews
WHERE __time > CURRENT_TIMESTAMP - INTERVAL '24' HOUR
GROUP BY 1, 2
ORDER BY page_views DESC
LIMIT 20;

Federate: Enrich Real-Time Data with Business Context

-- Join Druid real-time events with PostgreSQL user segments and S3 product data
SELECT
  d.event_hour,
  c.user_segment,
  p.product_category,
  SUM(d.page_views) AS total_views,
  COUNT(DISTINCT d.user_id) AS unique_users,
  CASE
    WHEN c.user_segment = 'Enterprise' THEN ROUND(SUM(d.page_views) * 2.5, 2)
    WHEN c.user_segment = 'Pro' THEN ROUND(SUM(d.page_views) * 1.5, 2)
    ELSE ROUND(SUM(d.page_views) * 0.5, 2)
  END AS estimated_value
FROM (
  SELECT
    DATE_TRUNC('hour', __time) AS event_hour,
    user_id,
    page,
    COUNT(*) AS page_views
  FROM "druid-realtime".druid.pageviews
  WHERE __time > CURRENT_TIMESTAMP - INTERVAL '24' HOUR
  GROUP BY 1, 2, 3
) d
LEFT JOIN "postgres-crm".public.users c ON d.user_id = c.user_id
LEFT JOIN "s3-catalog".products.page_mappings p ON d.page = p.page_url
GROUP BY d.event_hour, c.user_segment, p.product_category
ORDER BY estimated_value DESC;

Druid handles the real-time event aggregation, PostgreSQL provides user context, S3 maps pages to products, and Dremio joins everything.

Build a Semantic Layer Over Real-Time Data

CREATE VIEW analytics.gold.realtime_engagement AS
SELECT
  DATE_TRUNC('hour', __time) AS event_hour,
  page,
  COUNT(*) AS page_views,
  COUNT(DISTINCT user_id) AS unique_visitors,
  ROUND(COUNT(*) * 1.0 / COUNT(DISTINCT user_id), 2) AS views_per_visitor,
  CASE
    WHEN COUNT(*) > 10000 THEN 'Trending'
    WHEN COUNT(*) > 1000 THEN 'Active'
    WHEN COUNT(*) > 100 THEN 'Normal'
    ELSE 'Low Traffic'
  END AS traffic_tier
FROM "druid-realtime".druid.pageviews
WHERE __time > CURRENT_TIMESTAMP - INTERVAL '7' DAY
GROUP BY 1, 2;

Navigate to the Catalog, click Edit (pencil icon), and Generate Wiki and Generate Tags. Create descriptions like "realtime_engagement: Hourly page view metrics from the real-time clickstream, classified by traffic tier."

AI-Powered Analytics on Real-Time Data

Dremio AI Agent

The AI Agent lets users ask questions about real-time event data in plain English. Instead of writing complex time-window SQL, a product manager asks "Which pages are trending in the last 6 hours?" or "What's the average engagement per visitor for enterprise users today?" The Agent reads your wiki descriptions and generates accurate SQL.

This is particularly valuable for Druid data because time-series queries can be complex — date truncation, windowing, and aggregation syntax varies. The AI Agent handles this complexity automatically.

Dremio MCP Server

The Dremio MCP Server connects Claude, ChatGPT, and other AI clients to your Dremio data:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs for your AI client
Connect using mcp.dremio.cloud/mcp/{project_id}

A marketing team lead can ask Claude "Show me our highest-traffic pages from Druid data in the last 24 hours, broken down by user segment" and get real-time insights without writing SQL.

AI SQL Functions

Use AI to classify and analyze real-time event patterns:

-- Classify page traffic patterns with AI
SELECT
  page,
  page_views,
  unique_visitors,
  AI_CLASSIFY(
    'Based on this web traffic pattern, classify the likely content type',
    'Page: ' || page || ', Views: ' || CAST(page_views AS VARCHAR) || ', Unique visitors: ' || CAST(unique_visitors AS VARCHAR) || ', Views per visitor: ' || CAST(views_per_visitor AS VARCHAR),
    ARRAY['Product Page', 'Blog Content', 'Landing Page', 'Documentation', 'Support']
  ) AS inferred_content_type
FROM analytics.gold.realtime_engagement
WHERE traffic_tier = 'Trending';

-- Generate real-time traffic summaries
SELECT
  event_hour,
  AI_GENERATE(
    'Write a brief traffic summary for this hour',
    'Hour: ' || CAST(event_hour AS VARCHAR) || ', Total Views: ' || CAST(SUM(page_views) AS VARCHAR) || ', Unique Visitors: ' || CAST(SUM(unique_visitors) AS VARCHAR) || ', Trending Pages: ' || CAST(COUNT(CASE WHEN traffic_tier = 'Trending' THEN 1 END) AS VARCHAR)
  ) AS hourly_summary
FROM analytics.gold.realtime_engagement
GROUP BY event_hour
ORDER BY event_hour DESC
LIMIT 24;

Accelerate with Reflections

For historical aggregations over Druid data, create Reflections:

Build a view that aggregates Druid data by day/hour/week
Navigate to the view in the Catalog and click the Reflections tab
Choose Raw Reflection or Aggregation Reflection
Select columns and set the Refresh Interval — for real-time data, hourly; for historical trends, daily
Click Save

Dashboard queries for "last 30 days" or "year-over-year" hit the Reflection instead of scanning Druid's cold segments. Real-time queries for "last hour" still go directly to Druid for sub-second latency.

Governance on Real-Time Data

Druid has basic authentication but no column masking or row-level filtering. Dremio's Fine-Grained Access Control (FGAC) adds these capabilities:

Column masking: Mask user IDs, IP addresses, and location data from specific roles. A product manager sees engagement metrics but not individual user data.
Row-level filtering: Restrict real-time data access by team or region. A regional marketing team sees only their region's clickstream.
Unified policies: Same governance applies across Druid, PostgreSQL, S3, and all other sources.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Tableau: Dremio connector for direct Arrow Flight access to real-time dashboards
Power BI: Dremio ODBC driver or native connector
Python/Pandas: pyarrow.flight for programmatic access to event data
Looker: Connect via JDBC
dbt: dbt-dremio for SQL-based transformations on event data

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Druid data from their IDE. Ask Copilot "Show me trending pages from Druid in the last 6 hours" and get SQL generated from your semantic layer.

When to Keep Data in Druid vs. Migrate

Keep in Druid: Real-time event streams that need sub-second query latency, high-ingestion-rate data (thousands of events per second), data that powers real-time operational dashboards with sub-second SLAs.

Migrate to Iceberg: Historical event archives older than 30-90 days, data that needs SQL joins (Druid can't do them natively), analytics that combine events with dimensional data, data consumed by BI tools that expect standard SQL, archival data for compliance and auditing.

For active Druid data, create manual Reflections with refresh schedules that balance freshness and performance. For migrated Iceberg data in Dremio's Open Catalog, you get automated compaction, Autonomous Reflections, and significantly lower storage costs.

Real-Time Tiering Strategy

Combine Druid's real-time capabilities with Dremio's historical analysis:

Tier 1: Real-Time (Druid — 0 to 24 hours)

Druid ingests and serves sub-second queries on live event data. Dremio queries Druid directly for "last hour" or "last 6 hours" dashboards.

Tier 2: Recent Historical (Iceberg — 1 to 90 days)

Daily batch jobs move yesterday's data from Druid into Iceberg tables in Dremio's Open Catalog. Analytical queries for "last 30 days" hit Iceberg tables with Autonomous Reflections.

Tier 3: Long-Term Archive (Iceberg — 90+ days)

Older data stays in Iceberg cold storage (S3 Infrequent Access). Compliance and audit queries use time travel against archived snapshots.

-- Dremio view that combines real-time and historical data
CREATE VIEW analytics.gold.unified_events AS
SELECT event_type, user_id, event_timestamp, 'real-time' AS data_tier
FROM "druid-cluster".clickstream.events
WHERE event_timestamp >= CURRENT_TIMESTAMP - INTERVAL '24' HOUR
UNION ALL
SELECT event_type, user_id, event_timestamp, 'historical' AS data_tier
FROM analytics.silver.events_archive
WHERE event_timestamp < CURRENT_TIMESTAMP - INTERVAL '24' HOUR
  AND event_timestamp >= CURRENT_TIMESTAMP - INTERVAL '90' DAY;

Event Pipeline Integration

Common Druid deployment patterns that work with Dremio:

Kafka → Druid → Dremio: Real-time events flow through Kafka into Druid. Dremio queries Druid for analytics and joins with slow-changing dimensional data from PostgreSQL or S3.
Kafka → Druid + S3: Events land in both Druid (real-time) and S3 (archive). Dremio queries both seamlessly through federation.
Kinesis → Druid → Dremio: AWS-native pattern where Kinesis streams feed Druid, and Dremio provides multi-source analytics over streamed data.

Get Started

Apache Druid users can extend their real-time analytics with cross-source joins, AI-powered insights, enterprise governance, and Reflection-based acceleration — all through Dremio Cloud.

Try Dremio Cloud free for 30 days and connect your Druid cluster.

Connect MongoDB to Dremio Cloud: SQL Analytics on Document Data

Alex Merced — Sun, 01 Mar 2026 22:00:00 GMT

MongoDB is the most popular NoSQL document database. It stores data in flexible JSON-like documents, making it ideal for applications with evolving schemas — user profiles, product catalogs, IoT sensor data, and content management systems. But MongoDB's document model creates analytics challenges: you can't run SQL joins natively, aggregation pipelines are complex, and connecting MongoDB data to relational sources requires custom application code or ETL.

Dremio Cloud connects to MongoDB and exposes its collections as SQL-queryable tables. Nested documents appear as structured columns, and you can join MongoDB data with relational databases, data lakes, and cloud warehouses using standard SQL.

Why MongoDB Users Need Dremio

SQL on documents. MongoDB's query language (MQL) is powerful but different from SQL. Your analysts know SQL. Dremio transforms MongoDB collections into SQL-queryable tables, so analysts don't need to learn MQL or write aggregation pipelines.

Join documents with relational data. Your user profiles are in MongoDB, your order data is in PostgreSQL, and your marketing data is in S3. Without Dremio, combining these requires application code that queries each system separately and merges results in memory. Dremio federates all three in a single SQL query.

Flatten nested structures. MongoDB documents often contain nested objects and arrays. Dremio's FLATTEN function expands arrays into rows, and nested objects become addressable columns (e.g., address.city, preferences.theme).

Consistent governance. MongoDB has authentication and roles, but they don't extend to other data sources. Dremio's FGAC applies consistent column masking and row filtering across MongoDB and all other connected sources.

AI analytics. MongoDB's unstructured nature makes it difficult for AI tools to query directly. Dremio's semantic layer creates structured views with business context, enabling the AI Agent to answer questions about MongoDB data.

Prerequisites

MongoDB hostname or IP address (or MongoDB Atlas connection string)
Port — default 27017
Database name(s) — MongoDB databases you want to query
Username and password (with read role on target databases)
Network access — port 27017 open to Dremio Cloud. For MongoDB Atlas, add Dremio's IP range to the Atlas IP Access List
Dremio Cloud account — sign up free for 30 days

Connect MongoDB to Dremio Cloud

Click "+" and select MongoDB.
Enter Name, Host, Port (27017).
Authentication Type: Choose Standard (username/password) or No Authentication.
Configure Advanced Options:
- Use SSL: Enable for MongoDB Atlas or SSL-configured instances.
- Auth Database: The database used for authentication (default: admin).
- Read preference: Control whether queries hit primary or secondary replicas (primary, primaryPreferred, secondary, secondaryPreferred, nearest).
- Subpartition size: Controls how Dremio partitions large collections for parallel reads.
Configure Reflection Refresh and Metadata.
Set Privileges and Save.

Query MongoDB Data with SQL

-- Query a MongoDB collection as a SQL table
SELECT user_id, name, email, signup_date
FROM "mongo-users".app.users
WHERE signup_date > '2024-01-01'
ORDER BY signup_date DESC;

-- Access nested fields
SELECT
  user_id,
  name,
  address.city AS city,
  address.state AS state,
  preferences.theme AS ui_theme
FROM "mongo-users".app.users
WHERE address.state = 'CA';

Flatten Nested Arrays

MongoDB documents frequently contain arrays. Use FLATTEN to expand them into rows:

-- If each user document has an orders array
SELECT
  u.user_id,
  u.name,
  o.order_id,
  o.total_amount,
  o.order_date
FROM "mongo-users".app.users u,
  FLATTEN(u.orders) AS t(o)
WHERE o.total_amount > 100
ORDER BY o.order_date DESC;

Federate MongoDB with Relational Sources

-- Join MongoDB user profiles with PostgreSQL orders and S3 analytics
SELECT
  m.name AS customer_name,
  m.address.city AS city,
  COUNT(pg.order_id) AS total_orders,
  SUM(pg.amount) AS total_spent,
  COUNT(s3.event_id) AS engagement_events
FROM "mongo-users".app.users m
LEFT JOIN "postgres-orders".public.orders pg ON m.user_id = pg.customer_id
LEFT JOIN "s3-events".clickstream.events s3 ON m.user_id = s3.user_id
GROUP BY m.name, m.address.city
ORDER BY total_spent DESC;

Build a Semantic Layer

CREATE VIEW analytics.gold.customer_profile AS
SELECT
  m.user_id,
  m.name,
  m.email,
  m.address.city AS city,
  m.address.state AS state,
  m.signup_date,
  CASE
    WHEN m.subscription.tier = 'premium' THEN 'Premium'
    WHEN m.subscription.tier = 'pro' THEN 'Pro'
    ELSE 'Free'
  END AS subscription_tier
FROM "mongo-users".app.users m;

Navigate to the Catalog, click Edit (pencil icon), go to the Details tab, and click Generate Wiki and Generate Tags. Dremio's generative AI samples the view schema and data to produce descriptions like: "customer_profile: Contains one row per user combining profile data from MongoDB with subscription tier classification." Review and refine these descriptions — add business context like "Premium subscribers qualify for the dedicated support tier and priority feature access."

These wikis and labels are the context that powers Dremio's AI capabilities.

AI-Powered Analytics on MongoDB Data

MongoDB's flexible document model makes it notoriously difficult for AI tools to query directly — nested objects, variable schemas, and BSON types create barriers. Dremio's semantic layer solves this by creating structured, well-documented views over MongoDB data that AI tools can understand and query accurately.

Dremio AI Agent

The AI Agent lets business users ask questions about MongoDB data in plain English. Instead of learning MongoDB's aggregation framework or SQL with nested field syntax, a product manager asks "How many Premium subscribers are in California?" and the Agent generates the correct SQL using your semantic layer.

The Agent reads the wiki descriptions you attached to views to understand what "Premium" means in your data (subscription.tier = 'premium'), what "California" maps to (address.state = 'CA'), and which view to query. Better wikis produce more accurate AI responses.

Dremio MCP Server

The Dremio MCP Server extends AI capabilities to external chat clients. Connect Claude or ChatGPT to your MongoDB data through the hosted MCP Server with OAuth authentication:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs (e.g., https://claude.ai/api/mcp/auth_callback for Claude, https://chatgpt.com/connector_platform_oauth_redirect for ChatGPT)
Connect using mcp.dremio.cloud/mcp/{project_id} (US) or mcp.eu.dremio.cloud/mcp/{project_id} (EU)

Now your team can ask Claude "Show me user growth trends by subscription tier from MongoDB data" and get governed, accurate results — without knowing MongoDB query syntax or SQL.

AI SQL Functions

Use Dremio's built-in AI SQL functions to enrich MongoDB data directly in queries:

-- Classify users based on their MongoDB profile data
SELECT
  name,
  subscription_tier,
  city,
  state,
  AI_CLASSIFY(
    'Based on this user profile, classify their likely engagement level',
    'Name: ' || name || ', Subscription: ' || subscription_tier || ', City: ' || city || ', State: ' || state,
    ARRAY['Highly Engaged', 'Active', 'At Risk', 'Churned']
  ) AS engagement_prediction
FROM analytics.gold.customer_profile
WHERE subscription_tier IN ('Premium', 'Pro');

-- Generate personalized outreach messages
SELECT
  name,
  subscription_tier,
  AI_GENERATE(
    'Write a one-sentence personalized upgrade message for this user',
    'User: ' || name || ', Current Tier: ' || subscription_tier || ', Location: ' || city || ', ' || state
  ) AS upgrade_message
FROM analytics.gold.customer_profile
WHERE subscription_tier = 'Free';

AI_CLASSIFY categorizes users based on profile attributes. AI_GENERATE creates personalized text. Both run inline in your SQL queries, enriching MongoDB data with AI in real time.

Accelerate MongoDB Analytics with Reflections

MongoDB isn't designed for heavy analytical workloads. Running 50 dashboard queries per hour against MongoDB competes with your application's read/write operations. Create Reflections on your MongoDB views to cache results:

Navigate to the view in the Catalog
Create a Reflection with the columns and aggregations used most
Set the refresh interval (e.g., every 30 minutes for near-real-time, hourly for daily reporting)

BI tools connected to Dremio via Arrow Flight or ODBC get sub-second response times from Reflections — MongoDB handles zero analytical load.

MongoDB-Specific Considerations

Schema sampling. MongoDB is schema-less — each document can have different fields. Dremio samples documents to infer the schema. If your documents have highly variable schemas, some fields might not appear until more documents are sampled. You can increase the sample size in the source configuration.

Read preference. For MongoDB replica sets, use secondaryPreferred to route analytical queries to secondary replicas, avoiding impact on your primary node's CRUD operations.

Data types. MongoDB's BSON types map to Dremio types: ObjectID → VARCHAR, NumberLong → BIGINT, NumberInt → INT, Date → TIMESTAMP. Nested objects become structured columns addressable with dot notation. Arrays can be flattened with FLATTEN.

MongoDB Atlas. Add Dremio Cloud's IP range to your Atlas IP Access List. Enable SSL in the Dremio connection settings. Use the standard connection string hostname (not the SRV hostname).

When to Keep Data in MongoDB vs. Migrate to Iceberg

Keep in MongoDB: Data your application actively reads and writes, documents with evolving schemas that benefit from MongoDB's flexibility, operational data where real-time updates matter, data where document-level transactions are important.

Migrate to Iceberg: Historical user data, analytics-heavy aggregations, datasets that need SQL joins with relational sources, time-series data you query in aggregate, data consumed primarily by BI tools or AI agents.

For data that stays in MongoDB, create manual Reflections with refresh schedules matching your data freshness needs. This offloads analytical load from MongoDB while keeping data current. For migrated Iceberg data, Dremio provides automated compaction, time travel, results caching, and Autonomous Reflections.

Governance on MongoDB Data

MongoDB has database-level and collection-level access control, but no column masking or row-level filtering. Dremio's Fine-Grained Access Control (FGAC) adds these capabilities:

Column masking: Mask user emails, phone numbers, or payment details from specific roles. A product analyst sees user behavior patterns but not PII.
Row-level filtering: Restrict data by user role. A regional team sees only their region's user data.
Unified policies: Same governance applies across MongoDB, PostgreSQL, S3, Snowflake, and all other sources.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Tableau: Dremio connector — turns MongoDB documents into tabular data for Tableau
Power BI: Dremio ODBC driver or native connector
Python/Pandas: pyarrow.flight for programmatic access to flattened MongoDB data
Looker: Connect via JDBC
dbt: dbt-dremio for SQL-based transformations on MongoDB data

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query MongoDB data from their IDE. Ask Copilot "Show me user signup trends from MongoDB" and get SQL generated using your semantic layer — no aggregation pipeline knowledge needed.

Schema Flattening and Nested Documents

MongoDB stores data as nested JSON documents. Dremio automatically converts nested structures into queryable columns:

Top-level fields map directly to columns (name, email, created_at)
Nested objects use dot notation (address.city, address.state)
Arrays can be flattened using FLATTEN() to create one row per array element

-- Flatten nested order items from MongoDB documents
SELECT
  o.customer_id,
  o.order_date,
  f.item_name,
  f.quantity,
  f.unit_price
FROM "mongodb-app".ecommerce.orders o,
LATERAL FLATTEN(o.items) AS f(item_name, quantity, unit_price);

This SQL approach is simpler than MongoDB's aggregation pipeline ($unwind, $lookup, $group) for most analytical queries.

Dremio vs. MongoDB Atlas Data Federation

MongoDB Atlas Data Federation provides SQL-like access to MongoDB data. Key differences:

Feature	Dremio Cloud	Atlas Data Federation
Cross-source joins	PostgreSQL, S3, Snowflake, etc.	MongoDB + S3 only
Reflections	✅ Cache results	❌ Query every time
AI Agent	✅ Natural language queries	❌
Governance	Column masking + row filtering	MongoDB role-based access
BI connectivity	Arrow Flight (10-100x faster)	ODBC/JDBC only
Semantic layer	Views with wiki + tags	❌

Dremio provides a broader analytical platform, while Atlas Data Federation is specific to the MongoDB ecosystem.

Document-to-Analytics Pipeline

Optimize how MongoDB data flows into analytics:

Source layer: Dremio reads MongoDB collections directly — no ETL
Flattened views: Create SQL views that flatten nested documents into tabular format
Enrichment: Join flattened MongoDB data with relational and data lake sources
Semantic layer: Create business-ready views with wiki descriptions for AI

This pipeline runs entirely in SQL, eliminating the need for custom Python/Node.js ETL scripts to extract and transform MongoDB data.

Get Started

MongoDB users can query their document data with SQL, flatten nested structures, join MongoDB with relational databases and data lakes, and enable AI analytics — all without ETL pipelines or learning MongoDB's aggregation framework.

Try Dremio Cloud free for 30 days and connect your MongoDB instances.

Connect Vertica to Dremio Cloud: Federation for Analytics-Optimized Data

Alex Merced — Sun, 01 Mar 2026 21:00:00 GMT

Vertica is a columnar analytics database engineered for fast aggregate queries on large datasets. It was built from the ground up for analytical workloads — column-oriented storage, massively parallel processing, and automatic database design optimization. Organizations running Vertica typically have years of investment in analytics infrastructure: curated schemas, optimized projections, and sophisticated workloads that depend on Vertica's high-performance query engine.

But Vertica has limitations that become more painful as data ecosystems grow. Licensing costs scale with data volume. Federation with non-Vertica sources requires complex ETL. And connecting Vertica data to modern cloud tools, AI platforms, and cross-cloud architectures requires exporting data or building custom connectors.

Dremio Cloud connects to Vertica and queries it alongside your other data sources. Dremio's predicate pushdowns leverage Vertica's columnar engine for filtering and aggregation, while Reflections cache results to reduce ongoing Vertica compute load. You keep Vertica for what it does well and extend its reach to every other system in your organization.

Why Vertica Users Need Dremio

Reduce Vertica License Costs

Vertica's licensing model ties cost to data volume and node count. Every analytical query consumes cluster resources. As your data grows and more teams want access, the cost of scaling Vertica becomes significant. Dremio's Reflections provide an alternative: pre-compute the results of your most common queries and serve them from Dremio's cache instead of hitting Vertica on every request. Dashboard queries, scheduled reports, and ad-hoc exploration can all be served from Reflections, reducing the compute pressure on your Vertica cluster.

Federate with Cloud Sources

Vertica excels at analytical queries on its own data, but your organization's data lives in many places: S3 data lakes, PostgreSQL application databases, Snowflake cloud warehouses, MongoDB document stores. Without a federation layer, combining these with Vertica data requires ETL pipelines that extract from each source, transform, and load into Vertica. Dremio queries each source in place and joins the results — no data movement needed.

Modernize Without a Big-Bang Migration

Migrating away from Vertica is a large, risky project. Dremio lets you gradually shift analytical workloads. Start by querying Vertica through Dremio alongside new cloud-native sources (Apache Iceberg tables, S3 data lakes). As confidence grows, migrate specific datasets from Vertica to Iceberg tables in Dremio's Open Catalog, where they benefit from automated maintenance and lower storage costs. The migration happens incrementally, and Vertica continues serving critical workloads throughout.

Unified Governance

Vertica has its own access control, but it doesn't extend to your other data sources. Dremio's Fine-Grained Access Control applies consistent column masking and row-level filtering across Vertica, PostgreSQL, S3, and every other connected source from a single governance layer.

Prerequisites

Vertica hostname or IP address — the coordinator node of your Vertica cluster
Port — Vertica defaults to 5433
Database name
Username and password — a Vertica user with SELECT privileges on the tables you want to query
Network access — port 5433 must be reachable from Dremio Cloud
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Vertica to Dremio Cloud

1. Add the Vertica Source

In the Dremio console, click the "+" button in the left sidebar and select Vertica from the database source types.

2. Configure General Settings

Name: A descriptive identifier (e.g., analytics-vertica or web-analytics). This name appears in SQL queries as the source prefix.
Host: Your Vertica coordinator host.
Port: Default 5433.
Database: The Vertica database name.

3. Set Authentication

Provide the username and password for a Vertica user with read access. You can also use Secret Resource URL for password management through AWS Secrets Manager.

4. Configure Advanced Options

Setting	Purpose	Default
Record fetch size	Rows per batch from Vertica	200
Maximum Idle Connections	Idle connection pool size	8
Connection Idle Time (s)	Seconds before idle connections close	60
Connection Properties	Custom JDBC parameters	None

5. Set Reflection and Metadata Refresh

Configure how often Reflections refresh (re-query Vertica) and how often Dremio checks for new tables or schema changes. Click Save.

Query Vertica Data from Dremio

SELECT device_type, COUNT(*) AS sessions, AVG(session_duration_seconds) AS avg_duration, 
  SUM(page_views) AS total_page_views
FROM "analytics-vertica".web.sessions
WHERE session_date >= '2024-01-01' AND session_date < '2024-07-01'
GROUP BY device_type
ORDER BY sessions DESC;

Dremio pushes the date filter and aggregation to Vertica's columnar engine, which processes them efficiently against its compressed, column-oriented storage.

Federate Vertica with Other Sources

-- Join Vertica web analytics with PostgreSQL CRM and S3 marketing data
SELECT
  c.customer_segment,
  COUNT(v.session_id) AS total_sessions,
  AVG(v.session_duration_seconds) AS avg_session_duration,
  COUNT(DISTINCT v.user_id) AS unique_visitors,
  SUM(s3.ad_spend) AS marketing_spend,
  ROUND(COUNT(v.session_id) / NULLIF(SUM(s3.ad_spend), 0) * 1000, 2) AS sessions_per_thousand_dollars
FROM "analytics-vertica".web.sessions v
JOIN "postgres-crm".public.customers c ON v.user_id = c.customer_id
LEFT JOIN "s3-marketing".campaigns.spend_by_segment s3 ON c.customer_segment = s3.segment
WHERE v.session_date >= '2024-01-01'
GROUP BY c.customer_segment
ORDER BY total_sessions DESC;

Vertica handles the session aggregation, PostgreSQL handles the customer lookup, and Dremio handles the cross-source join.

Build a Semantic Layer

CREATE VIEW analytics.gold.web_performance AS
SELECT
  v.device_type,
  v.session_date,
  COUNT(*) AS sessions,
  AVG(v.session_duration_seconds) AS avg_duration_seconds,
  SUM(v.page_views) AS total_page_views,
  SUM(CASE WHEN v.converted = true THEN 1 ELSE 0 END) AS conversions,
  ROUND(SUM(CASE WHEN v.converted = true THEN 1 ELSE 0 END) * 100.0 / COUNT(*), 2) AS conversion_rate_pct,
  CASE
    WHEN AVG(v.session_duration_seconds) > 300 THEN 'High Engagement'
    WHEN AVG(v.session_duration_seconds) > 120 THEN 'Moderate Engagement'
    ELSE 'Low Engagement'
  END AS engagement_tier
FROM "analytics-vertica".web.sessions v
GROUP BY v.device_type, v.session_date;

Navigate to the Catalog, click Edit (pencil icon) on the view, and Generate Wiki and Generate Tags. This creates the business context that powers AI features.

AI-Powered Analytics on Vertica Data

Dremio AI Agent

The built-in AI Agent lets users ask questions about your Vertica data in plain English. Instead of writing complex analytical SQL, a marketing manager can ask "What's our conversion rate on mobile this quarter?" The Agent reads the wiki descriptions attached to your views, understands what "conversion rate" and "mobile" mean in your data, and generates the correct SQL.

The quality of the AI Agent's responses depends directly on the quality of your semantic layer. Wikis that explain "conversion_rate_pct is the percentage of web sessions that resulted in a purchase" produce better results than technical column names alone.

Dremio MCP Server

The Dremio MCP Server extends AI capabilities to external chat clients. Connect Claude or ChatGPT to your Dremio data through the hosted MCP Server with OAuth authentication:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs for your AI client
Connect using mcp.dremio.cloud/mcp/{project_id}

Now your team can ask Claude "Analyze our web engagement trends from Vertica data this quarter" and get accurate, governed results — without writing SQL or accessing Vertica directly.

AI SQL Functions

Use AI SQL functions directly in queries to enrich Vertica data:

-- Classify web sessions by potential value
SELECT
  session_id,
  device_type,
  page_views,
  session_duration_seconds,
  AI_CLASSIFY(
    'Based on this browsing behavior, classify the user intent',
    'Device: ' || device_type || ', Pages: ' || CAST(page_views AS VARCHAR) || ', Duration: ' || CAST(session_duration_seconds AS VARCHAR) || 's',
    ARRAY['Purchase Intent', 'Research', 'Browsing', 'Bounced']
  ) AS predicted_intent
FROM "analytics-vertica".web.sessions
WHERE session_date = CURRENT_DATE;

AI_CLASSIFY runs LLM inference inside your SQL query, classifying each web session from Vertica data into intent categories. AI_GENERATE can produce narrative summaries, and AI_SIMILARITY can find semantic matches between text fields.

Accelerate Vertica Queries with Reflections

Create Reflections on your most frequently queried views:

Navigate to the view in the Catalog
Click the Reflections tab
Choose Raw Reflection (full cache) or Aggregation Reflection (pre-computed metrics)
Select columns and aggregations
Set the Refresh Interval — for Vertica data that updates daily, daily refresh works; for real-time dashboards, match the refresh to your SLA
Click Save

BI tools connected via Arrow Flight or ODBC get sub-second response times from Reflections, even though the underlying data lives in Vertica. A conversion analytics dashboard that queries Vertica 96 times per day with a daily Reflection refresh consumes Vertica resources only once — a 99% reduction in cluster load.

Governance Across Vertica and Other Sources

Dremio's Fine-Grained Access Control (FGAC) provides governance that extends beyond Vertica to every connected source:

Column masking: Mask conversion rates, revenue data, or user identifiers from specific roles. A product manager sees engagement metrics but not raw revenue.
Row-level filtering: Restrict data visibility based on user roles. Regional teams see only their region's data automatically.
Unified policies: Same governance applies across Vertica, PostgreSQL, S3, BigQuery, and all other sources — no per-database policy management.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, MCP Server, and Arrow Flight clients.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Tableau: Dremio connector for direct Arrow Flight access
Power BI: Dremio ODBC driver or native connector
Python/Pandas: pyarrow.flight for programmatic data access to Vertica analytics
Looker: Connect via JDBC
dbt: dbt-dremio for SQL-based transformations

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Vertica data from their IDE. Ask Copilot "Show me conversion rates by device type from web analytics" and get SQL generated from your semantic layer — without switching to the Dremio console.

When to Keep Data in Vertica vs. Migrate to Iceberg

Keep in Vertica: Active analytical workloads optimized with Vertica projections, data with complex Vertica-specific features (database designer optimizations, flex tables), workloads that depend on Vertica's sub-second response times for real-time dashboards.

Migrate to Iceberg: Historical data and archives, datasets consumed by non-Vertica tools, data where Vertica licensing cost per TB exceeds the analytical value, datasets that benefit from time travel and automated compaction.

For data that stays in Vertica, create manual Reflections to reduce query load. For migrated data, Dremio's Open Catalog provides automated compaction, time travel, and Autonomous Reflections at a fraction of the per-TB cost.

Vertica Deployment Modes and Dremio

Vertica has two deployment modes, both compatible with Dremio:

Enterprise Mode (On-Premises)

Traditional deployment with local storage. Dremio connects via JDBC and pushes SQL operations to Vertica's engine when possible. Reflections are particularly valuable here — they offload analytical queries and reduce the on-premises compute needed.

EON Mode (Cloud-Optimized)

Vertica's compute-storage separation architecture on AWS, Azure, or GCP. Dremio connects the same way, but EON mode's elastic compute makes Reflections' cost-saving impact even more significant — when Dremio serves cached results, EON subclusters can scale down.

Vertica-Specific SQL Considerations

Dremio handles most Vertica SQL natively. For Vertica-specific syntax:

Projections: Vertica projections are transparent to Dremio — Vertica automatically uses optimal projections for queries pushed down
Flex tables: Dremio reads flex table columns as VARCHAR — cast to appropriate types in your Dremio views
COPY LOCAL: Not available through Dremio — use Dremio's own CREATE TABLE AS SELECT for data loading
Vertica ML functions: Use external queries for Vertica-specific ML functions: SELECT * FROM TABLE("vertica-analytics".EXTERNAL_QUERY('SELECT PREDICT_LINEAR...'))

Migration ROI Example

A mid-sized organization with 50TB in Vertica Enterprise:

Current cost: ~$500K/year in Vertica licensing (per-TB pricing)
Migrate 30TB of historical data to Iceberg: Eliminates 60% of licensed data volume
Remaining 20TB in Vertica: Active analytical workloads, protected by Reflections
Net result: Potential 40-60% reduction in Vertica licensing costs, with improved analytics capabilities (AI, federation, governance) on all data

Get Started

Vertica users can reduce licensing pressure, federate with cloud sources, modernize incrementally, and add AI analytics — all through Dremio Cloud. Connect your Vertica cluster to Dremio, create Reflections on your most-queried tables, and start tracking the reduction in Vertica query load as Dremio serves cached results.

Try Dremio Cloud free for 30 days and connect your Vertica cluster alongside your other data sources.

Connect Azure Synapse Analytics to Dremio Cloud: Multi-Cloud Data Warehouse Federation

Alex Merced — Sun, 01 Mar 2026 20:00:00 GMT

Microsoft Azure Synapse Analytics combines big data analytics and enterprise data warehousing into a single Azure-integrated platform. If your organization has chosen the Microsoft cloud ecosystem, your cleaned and modeled analytical data likely lives in Synapse dedicated SQL pools or serverless SQL pools. Synapse works well within Azure, but it creates challenges when you need to connect that data with AWS, Google Cloud, or on-premises databases. Azure Data Factory pipelines handle some of this, but they add cost, latency, and engineering complexity.

Dremio Cloud connects to Azure Synapse and federates it with every other data source in your organization. Synapse queries push down to Synapse's engine for processing, and Dremio handles cross-source joins, query acceleration with Reflections, unified governance, and AI-powered analytics. You keep your investment in Synapse while extending its reach beyond the Azure ecosystem.

Why Azure Synapse Users Need Dremio

Multi-Cloud Analytics Without Data Movement

Your Azure Synapse workspace holds curated sales and finance data, but your application database runs on Amazon RDS (PostgreSQL), your marketing attribution data is in Google BigQuery, and your raw event logs sit in Amazon S3. Without a federation layer, joining these datasets requires Azure Data Factory to extract data from non-Azure sources, transform it, and load it into Synapse — a process that can take hours and costs real money in compute and data egress.

Dremio eliminates this entirely. Connect Synapse, PostgreSQL, BigQuery, and S3 as separate sources in Dremio, and write a single SQL query that joins across all four. Dremio's query optimizer pushes filtering and aggregation to each source (predicate pushdown), transfers only the results, and handles the cross-source join in its own engine. No pipelines. No data movement.

Cost Optimization Through Reflections

Synapse dedicated SQL pools charge based on the Data Warehouse Units (DWUs) provisioned, and serverless pools charge per TB of data processed. Dashboard queries that run every 15 minutes, ad-hoc exploration by analysts, and scheduled reports all consume Synapse compute resources.

Dremio's Reflections create pre-computed materializations of your most frequently run queries. After the initial execution, subsequent queries that match the Reflection pattern are served from Dremio's cache — not from Synapse. This can reduce Synapse compute consumption by 50-80% for dashboard and reporting workloads, directly lowering your Azure bill.

Unified Governance Across Clouds

Azure Synapse has role-based access control and Azure Active Directory integration within the Azure ecosystem. But those policies don't extend to your AWS databases or Google Cloud data. Dremio's Fine-Grained Access Control (FGAC) applies consistent column masking (hiding Social Security numbers, email addresses) and row-level filtering (restricting data by region or department) across Synapse and every other connected source. One governance policy, applied everywhere.

The Semantic Layer for Business Context

Raw Synapse tables have technical column names and no business context. Dremio lets you create views that encapsulate business logic (what "active customer" or "quarterly revenue" means), then attach wiki descriptions and labels to those views. This semantic layer makes your data self-documenting and powers Dremio's AI capabilities.

Prerequisites

Before connecting Azure Synapse to Dremio Cloud, confirm you have:

Synapse SQL endpoint — the fully qualified server name from your Synapse workspace (e.g., myworkspace.sql.azuresynapse.net)
Port number — default 1433 (Synapse uses the same port as SQL Server)
Database name — the specific SQL pool (dedicated or serverless) you want to connect
Username and password — SQL authentication credentials with read access to the tables you want to query
Network access — Synapse's firewall must allow connections from Dremio Cloud's IP addresses. Configure this in the Synapse workspace's networking settings
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Azure Synapse to Dremio Cloud

1. Add the Synapse Source

In the Dremio console, click the "+" button in the left sidebar and select Microsoft Azure Synapse Analytics from the database source types. Alternatively, navigate to Databases and click Add database.

2. Configure General Settings

Name: A descriptive identifier for this source (e.g., synapse-analytics or azure-sales-warehouse). This name appears in your SQL queries as the source prefix. Cannot include /, :, [, or ].
Host: Your Synapse SQL endpoint (e.g., myworkspace.sql.azuresynapse.net).
Port: Default 1433.
Database: The SQL pool name you want to connect to.

3. Set Authentication

Master Credentials: Enter the SQL authentication username and password with SELECT permissions on the schemas and tables you want to query.
Secret Resource URL: Store the password in AWS Secrets Manager and provide the ARN. Dremio fetches the password at connection time for centralized credential management.

4. Configure Advanced Options

Setting	What It Does	Default
Record fetch size	Rows per batch from Synapse	200
Maximum Idle Connections	Idle connection pool size	8
Connection Idle Time (s)	Seconds before idle connections close	60
Encrypt Connection	Enable SSL/TLS encryption	Off
Connection Properties	Custom JDBC parameters	None

5. Set Reflection Refresh and Metadata

Reflection Refresh: How often Dremio re-queries Synapse to update cached materializations. For dashboards with hourly data, set to 1-4 hours. For stable reporting data, daily or weekly.
Metadata Refresh: How often Dremio checks for new tables or schema changes. Default 1 hour for discovery, 1 hour for details.

6. Set Privileges and Save

Optionally restrict which Dremio users or roles can access this Synapse source. Click Save to create the connection.

Query Azure Synapse Data from Dremio

Once connected, browse your Synapse schemas and tables in the SQL Runner:

SELECT region, product_line, SUM(revenue) AS total_revenue, COUNT(order_id) AS order_count
FROM "synapse-analytics".dbo.sales_summary
WHERE fiscal_year = 2024 AND region IN ('EMEA', 'APAC', 'Americas')
GROUP BY region, product_line
ORDER BY total_revenue DESC;

Dremio pushes the WHERE clause and aggregation to Synapse — only the summarized result crosses the network.

Federate Azure Synapse with Other Sources

The real power emerges when you combine Synapse data with non-Azure sources:

-- Join Synapse sales data with AWS-hosted CRM and S3 marketing data
SELECT
  syn.region,
  syn.product_line,
  syn.total_revenue AS synapse_revenue,
  pg.customer_count,
  s3.marketing_spend,
  ROUND(syn.total_revenue / NULLIF(s3.marketing_spend, 0), 2) AS revenue_per_marketing_dollar
FROM (
  SELECT region, product_line, SUM(revenue) AS total_revenue
  FROM "synapse-analytics".dbo.sales_summary
  WHERE fiscal_year = 2024
  GROUP BY region, product_line
) syn
LEFT JOIN (
  SELECT region, COUNT(DISTINCT customer_id) AS customer_count
  FROM "postgres-crm".public.customers
  GROUP BY region
) pg ON syn.region = pg.region
LEFT JOIN "s3-marketing".campaigns.regional_spend s3
  ON syn.region = s3.region
ORDER BY revenue_per_marketing_dollar DESC;

Three clouds (Azure, AWS, S3), one query, no ETL pipelines.

Build a Semantic Layer Over Synapse Data

Create views that translate technical Synapse schemas into business-friendly analytics:

CREATE VIEW analytics.gold.regional_performance AS
SELECT
  s.region,
  s.product_line,
  SUM(s.revenue) AS total_revenue,
  SUM(s.cost) AS total_cost,
  SUM(s.revenue) - SUM(s.cost) AS gross_profit,
  ROUND((SUM(s.revenue) - SUM(s.cost)) / NULLIF(SUM(s.revenue), 0) * 100, 1) AS profit_margin_pct,
  CASE
    WHEN SUM(s.revenue) > 1000000 THEN 'Major Market'
    WHEN SUM(s.revenue) > 250000 THEN 'Growth Market'
    ELSE 'Emerging Market'
  END AS market_tier
FROM "synapse-analytics".dbo.sales_summary s
WHERE s.fiscal_year = 2024
GROUP BY s.region, s.product_line;

Navigate to the Catalog, click Edit (pencil icon) on this view, go to the Details tab, and click Generate Wiki and Generate Tags. Dremio's generative AI samples the view schema and data to produce descriptions that help analysts and AI tools understand the dataset.

AI-Powered Analytics on Synapse Data

Dremio provides three AI capabilities that transform how you work with Synapse data:

Dremio AI Agent

The built-in AI Agent lets users ask questions about your Synapse data in plain English. Instead of writing SQL, a business user can ask "What's our profit margin by region?" and the AI Agent generates the correct SQL based on the semantic layer (wikis, labels, view definitions) you've built.

The AI Agent reads the wiki descriptions you attached to your views to understand what columns mean in business terms. This is why the semantic layer matters — better metadata produces more accurate AI-generated queries. For example, if your regional_performance view has a wiki that says "profit_margin_pct represents the gross profit margin after cost of goods sold," the Agent uses that context to correctly answer "Which regions are most profitable?"

Dremio MCP Server

The Dremio MCP Server extends AI capabilities beyond Dremio's own interface. It's an open-source project that enables AI chat clients like Claude and ChatGPT to securely interact with your Dremio data using natural language.

The Dremio-hosted MCP Server provides OAuth support, which guarantees user identity, authentication, and authorization for all interactions. Once connected, you can use natural language in Claude or ChatGPT to:

Explore your Synapse data schemas and tables
Run analytical queries and get results
Create visualizations from query results
Build and save views

Setup is straightforward:

Create a Native OAuth application in Dremio Cloud
Configure the redirect URLs for your AI chat client
Connect using the MCP endpoint: mcp.dremio.cloud/mcp/{project_id} (US) or mcp.eu.dremio.cloud/mcp/{project_id} (EU)

This means a marketing manager can ask Claude "Show me our top 5 regions by profit margin from the Synapse sales data" and get accurate, governed results — without knowing SQL or having direct Synapse access.

AI SQL Functions

Dremio provides built-in AI SQL functions that you can use directly in queries against any connected data, including Synapse:

-- Classify products based on their Synapse metadata
SELECT
  product_line,
  total_revenue,
  AI_CLASSIFY(
    'Based on this revenue and growth pattern, classify the product health',
    product_line || ': $' || CAST(total_revenue AS VARCHAR) || ' revenue',
    ARRAY['Thriving', 'Stable', 'Declining', 'At Risk']
  ) AS product_health
FROM "synapse-analytics".dbo.product_summary;

-- Generate summaries from Synapse data
SELECT
  region,
  AI_GENERATE(
    'Write a one-sentence business summary for this regional performance',
    'Region: ' || region || ', Revenue: $' || CAST(revenue AS VARCHAR) || ', Growth: ' || CAST(yoy_growth AS VARCHAR) || '%'
  ) AS executive_summary
FROM "synapse-analytics".dbo.regional_metrics;

These functions run LLM inference directly in your SQL queries, turning raw Synapse data into AI-enriched insights.

Accelerate Synapse Queries with Reflections

For queries that run repeatedly (dashboard refreshes, scheduled reports):

Build a view over your Synapse data (like regional_performance above).
In the Catalog, select the view and create a Reflection.
Choose the columns and aggregations to include.
Set the refresh interval (how often Dremio re-queries Synapse to update the Reflection).

After the Reflection is built, Dremio's query optimizer automatically routes matching queries to the Reflection. Your BI tools (Power BI, Tableau) connected via Arrow Flight or ODBC get sub-second responses from the Reflection instead of waiting for Synapse to process the query. The acceleration is completely transparent — users write the same SQL and see the same data, just faster.

When to Keep Data in Synapse vs. Migrate to Iceberg

Keep in Synapse: Data actively consumed by Azure-native tools (Power BI with DirectQuery, Azure Machine Learning), data with complex Synapse-specific transformations, data shared through Azure Data Share.

Migrate to Iceberg: Historical archive data that's rarely updated, large analytical datasets that would benefit from automated compaction and manifest optimization, datasets that need time travel (query as of any past timestamp), data that other teams access through non-Azure tools.

For data that stays in Synapse, create manual Reflections with refresh schedules matching your data freshness requirements. For migrated Iceberg data, Dremio's Open Catalog provides automated compaction, time travel, and Autonomous Reflections.

Governance Across Azure Synapse and Other Sources

Dremio's Fine-Grained Access Control (FGAC) extends Synapse's Azure AD-based security to every connected source:

Column masking: Mask revenue, cost, and margin data from specific roles. A marketing analyst sees conversion counts but not financial details.
Row-level filtering: Regional managers see only their region's data automatically across all sources.
Unified policies: Same governance applies to Synapse, PostgreSQL, S3, BigQuery, and all other sources — no per-service security configuration.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, MCP Server, and Arrow Flight clients.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Power BI: Dremio native connector — ideal for Azure-centric organizations
Tableau: Dremio connector for direct Arrow Flight access
Python/Pandas: pyarrow.flight for programmatic data access
Looker: Connect via JDBC
dbt: dbt-dremio adapter for SQL-based transformations

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Synapse data from their IDE. Ask Copilot "Show me regional profit margins from Azure Synapse" and get SQL generated from your semantic layer.

Get Started

Azure Synapse users can extend their warehouse beyond the Azure ecosystem, reduce compute costs with Reflections, and enable AI-powered analytics across all their data sources — all through Dremio Cloud.

Try Dremio Cloud free for 30 days and connect your Azure Synapse workspace alongside your other data sources.

Connect Snowflake to Dremio Cloud: Federate, Govern, and Accelerate Beyond Snowflake

Alex Merced — Sun, 01 Mar 2026 19:00:00 GMT

Snowflake is a popular cloud data warehouse known for its separation of storage and compute, near-zero maintenance, and broad ecosystem. Many organizations have made Snowflake their primary analytics platform. But as data ecosystems mature, limitations emerge: Snowflake credits are consumed on every query, connecting Snowflake data to non-Snowflake sources requires data sharing agreements or ETL, and running all workloads in Snowflake means paying Snowflake prices for everything — including repetitive dashboard queries and ad-hoc exploration.

Dremio Cloud connects to Snowflake as a federated data source. You can query Snowflake tables directly, join them with PostgreSQL, S3, MongoDB, BigQuery, and any other connected source in a single SQL query, and accelerate repeated queries with Reflections so they don't burn Snowflake credits on every execution.

Snowflake's native Iceberg Tables feature allows managing Iceberg-formatted data within Snowflake. However, this still keeps your compute costs within Snowflake's pricing model. By combining Dremio Cloud with Snowflake (and potentially Snowflake's Open Catalog for shared Iceberg access), organizations can use Snowflake for data engineering while leveraging Dremio for cost-optimized analytical serving. This hybrid approach gives you Snowflake's data engineering strengths without paying Snowflake credit rates for every analytical query.

The cost concern is real: organizations regularly report that 40-60% of their Snowflake spend comes from dashboards, scheduled reports, and ad-hoc queries — workloads that are fundamentally repetitive and ideal for Reflection-based caching.

Why Snowflake Users Need Dremio

Reduce Snowflake Credit Consumption

Every query in Snowflake consumes credits based on the warehouse size and query runtime. Dashboard queries that run every 15 minutes, analytics training sessions, ad-hoc data exploration by 50 analysts, and nightly scheduled reports all consume credits.

Dremio's Reflections create pre-computed materializations of frequently executed queries. After the initial run, matching queries are served from Dremio's cache instead of Snowflake. For organizations spending over $100K/year on Snowflake compute, routing read-heavy analytical and dashboard workloads through Dremio can reduce credit consumption by 30-70% on those workloads.

Federation Beyond Snowflake

Snowflake's data sharing works between Snowflake accounts. But what about your PostgreSQL application database, your S3 data lake, your MongoDB user profiles, or your on-premises Oracle ERP? Joining these with Snowflake data requires ETL pipelines — extracting from each source, transforming, and loading into Snowflake. Dremio queries each source in place and joins the results in its own engine. No data movement, no Snowflake ingestion costs.

Unified Governance

Snowflake has robust access controls within Snowflake. But governing data across Snowflake, PostgreSQL, S3, and MongoDB requires separate policies in each system. Dremio's Fine-Grained Access Control applies consistent column masking and row-level filtering across all connected sources from a single interface.

AI Analytics Across All Sources

Snowflake has AI/ML features within its ecosystem (Cortex). Dremio adds AI capabilities that span your entire data estate, not just Snowflake — including an AI Agent for natural language queries, an MCP Server for external AI tools, and SQL-level AI functions.

Prerequisites

Snowflake account URL (e.g., myaccount.snowflakecomputing.com)
Username and password (or OAuth/key pair authentication)
Warehouse name — the compute resource Snowflake uses for queries
Database name — the Snowflake database you want to connect
Network access from Dremio Cloud to your Snowflake instance
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Snowflake to Dremio Cloud

1. Add the Snowflake Source

In the Dremio console, click "+" in the left sidebar and select Snowflake.

2. Configure Connection Details

Name: A descriptive identifier (e.g., snowflake-warehouse or analytics-snowflake).
Account URL: Your Snowflake account URL.
Warehouse: The Snowflake virtual warehouse to use for queries.
Database: The Snowflake database to connect to.

3. Set Authentication

Choose from Master Credentials (username/password), OAuth, or key pair authentication.

4. Configure Advanced Settings

Setting	Purpose
Record fetch size	Rows per batch from Snowflake
Maximum Idle Connections	Connection pool management
Connection Properties	Custom Snowflake connection parameters

5. Set Reflection and Metadata Refresh

Configure how often Reflections refresh and how often Dremio checks for schema changes. Click Save.

Query Snowflake Data from Dremio

SELECT
  product_category,
  SUM(sales_amount) AS total_sales,
  COUNT(DISTINCT customer_id) AS unique_buyers,
  ROUND(SUM(sales_amount) / COUNT(DISTINCT customer_id), 2) AS avg_spend_per_customer
FROM "snowflake-warehouse".PUBLIC.SALES_FACT
WHERE sale_date >= '2024-01-01' AND sale_date < '2024-07-01'
GROUP BY product_category
ORDER BY total_sales DESC;

Federate Snowflake with Other Sources

-- Join Snowflake sales with PostgreSQL reviews and S3 return data
SELECT
  sf.product_category,
  sf.total_sales,
  sf.unique_buyers,
  pg.avg_review_score,
  pg.review_count,
  s3.return_rate,
  ROUND(sf.total_sales * (1 - s3.return_rate), 2) AS net_revenue
FROM (
  SELECT product_category, SUM(sales_amount) AS total_sales, COUNT(DISTINCT customer_id) AS unique_buyers
  FROM "snowflake-warehouse".PUBLIC.SALES_FACT
  WHERE sale_date >= '2024-01-01'
  GROUP BY product_category
) sf
LEFT JOIN "postgres-reviews".public.product_reviews pg ON sf.product_category = pg.category
LEFT JOIN "s3-analytics".returns.category_return_rates s3 ON sf.product_category = s3.category
ORDER BY net_revenue DESC;

Build a Semantic Layer

CREATE VIEW analytics.gold.product_health AS
SELECT
  sf.product_category,
  SUM(sf.sales_amount) AS total_revenue,
  COUNT(DISTINCT sf.customer_id) AS unique_customers,
  ROUND(SUM(sf.sales_amount) / COUNT(DISTINCT sf.customer_id), 2) AS customer_value,
  CASE
    WHEN SUM(sf.sales_amount) > 1000000 THEN 'Category Leader'
    WHEN SUM(sf.sales_amount) > 250000 THEN 'Growth Category'
    ELSE 'Emerging'
  END AS category_tier
FROM "snowflake-warehouse".PUBLIC.SALES_FACT sf
GROUP BY sf.product_category;

Click Edit (pencil icon) in the Catalog, then Generate Wiki and Generate Tags to create AI-readable business context.

AI-Powered Analytics on Snowflake Data

Dremio AI Agent

Users ask questions in plain English: "Which product categories are growing fastest?" The AI Agent reads your wiki descriptions and generates accurate SQL. The semantic layer you've built is the foundation — better descriptions mean better AI responses.

Dremio MCP Server

The Dremio MCP Server connects external AI tools (Claude, ChatGPT) to your Dremio data with OAuth authentication:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs for your AI client
Connect using mcp.dremio.cloud/mcp/{project_id}

A product manager can ask ChatGPT "What are our top 5 product categories by net revenue from Snowflake?" and get governed, accurate results.

AI SQL Functions

-- Generate product insights with AI
SELECT
  product_category,
  total_revenue,
  customer_value,
  AI_GENERATE(
    'Write a one-sentence product strategy recommendation',
    'Category: ' || product_category || ', Revenue: $' || CAST(total_revenue AS VARCHAR) || ', Customer Value: $' || CAST(customer_value AS VARCHAR) || ', Tier: ' || category_tier
  ) AS strategy_recommendation
FROM analytics.gold.product_health;

-- Classify product categories
SELECT
  product_category,
  AI_CLASSIFY(
    'Based on these metrics, classify the investment priority',
    'Revenue: $' || CAST(total_revenue AS VARCHAR) || ', Customers: ' || CAST(unique_customers AS VARCHAR),
    ARRAY['High Priority', 'Medium Priority', 'Low Priority', 'Divest']
  ) AS investment_priority
FROM analytics.gold.product_health;

Accelerate with Reflections

Create Reflections on frequently queried Snowflake views to offload repeated queries from Snowflake credits:

In the Catalog, navigate to the view you want to accelerate
Click the Reflections tab
Choose Raw Reflection (full dataset cache) or Aggregation Reflection (pre-computed SUM/COUNT/AVG)
Select the columns and aggregations to include
Set the Refresh Interval — balance between data freshness and Snowflake credit consumption
Click Save

After creation, Dremio's query optimizer automatically routes matching queries to the Reflection. Dashboard queries and scheduled reports hit the cache instead of consuming Snowflake credits. BI tools connected via Arrow Flight get sub-second response times.

Example: Dashboard Acceleration

A Tableau dashboard that refreshes every 15 minutes queries product_health. Without Reflections, that's 96 Snowflake queries per day. With a Reflection that refreshes every 2 hours, Dremio serves 84 of those queries from cache — an 87.5% reduction in Snowflake credit consumption for that dashboard alone. Multiply that across 50 dashboards and the savings become significant.

Governance Across Snowflake and Other Sources

Dremio's Fine-Grained Access Control (FGAC) provides governance capabilities that work across Snowflake and every other connected source:

Column masking: Mask sensitive customer data (PII, financial details) from specific user roles. A marketing analyst sees customer_name but not social_security_number. An auditor sees both.
Row-level filtering: Automatically filter data based on the querying user's role. A regional manager sees only their region's data across all sources.
Unified policies: The same governance rules apply whether data comes from Snowflake, PostgreSQL, S3, or any other source — no per-source policy management.

These policies apply across all access methods: SQL Runner, BI tools, AI Agent, MCP Server, and Arrow Flight clients.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer compared to JDBC/ODBC. After building views over Snowflake data:

Tableau: Use the Dremio connector for direct Arrow Flight access
Power BI: Use Dremio's ODBC driver or native connector
Python/Pandas: Use pyarrow.flight for programmatic data access
Looker: Connect via JDBC with Dremio's driver
dbt: Use dbt-dremio for SQL-based transformations

All queries benefit from Reflections, governance, and semantic layer context.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Snowflake data from their IDE. Ask Copilot "Show me product health metrics from Snowflake" and it generates SQL using your semantic layer — without switching to the Dremio console or Snowflake's Worksheets.

External Queries

For Snowflake-specific functions not natively supported in Dremio's SQL, use external queries:

SELECT * FROM TABLE(
  "snowflake-warehouse".EXTERNAL_QUERY(
    'SELECT APPROX_COUNT_DISTINCT(customer_id), MEDIAN(sales_amount) FROM PUBLIC.SALES_FACT WHERE sale_date >= ''2024-01-01'''
  )
);

External queries pass raw SQL to Snowflake for execution, returning results through Dremio. This is useful for functions like APPROX_COUNT_DISTINCT, QUALIFY, or Snowflake-specific window functions.

When to Keep Data in Snowflake vs. Migrate

Keep in Snowflake: Data consumed by Snowflake-native tools (Snowpipe, Streams, Tasks), data shared through Snowflake Data Sharing, workloads with Snowflake-specific features (materialized views, dynamic tables), datasets actively managed by Snowflake-based ETL.

Migrate to Iceberg: Historical data that rarely changes, archival tables, datasets consumed primarily through non-Snowflake tools, workloads where Snowflake credit costs exceed the analytical value delivered. Migrated Iceberg tables benefit from Dremio's automatic compaction, time travel, Autonomous Reflections, and zero per-query storage costs.

For data that stays in Snowflake, create manual Reflections to reduce credit consumption. For migrated Iceberg data, Dremio handles optimization automatically.

Snowflake Credit Optimization with Dremio

Credit Consumption by Warehouse Size

Warehouse Size	Credits/Hour	Dremio Reflection Impact
X-Small	1	Reflections serve cached queries — warehouse suspends faster
Small	2	Same pattern — faster auto-suspend reduces credit burn
Medium	4	Dashboard workloads offloaded — downsize to Small
Large	8	Interactive + scheduled workloads offloaded — significant savings
X-Large	16	Heavy analytical workloads cached — potential 50%+ reduction

Quantifying Credit Savings

Example calculation for a medium-sized analytics team:

Without Dremio: 50 analysts + 20 dashboards consume ~$15,000/month in Snowflake credits
With Dremio Reflections: Dashboard queries (60% of total) served from cache → ~$6,000/month savings
Net impact: $9,000/month Snowflake bill + Dremio costs, typically netting 20-40% total savings

Snowflake Data Cloud Integration

Dremio doesn't replace Snowflake's Data Cloud capabilities — it complements them:

Data Sharing: Continue sharing datasets via Snowflake Data Sharing with other Snowflake accounts
Marketplace: Access Snowflake Marketplace datasets alongside your own, but federate them with non-Snowflake sources through Dremio
Snowpark: Continue using Snowpark for Python/Java/Scala processing within Snowflake
Dremio's role: Federation with non-Snowflake data, AI analytics, Reflection-based BI serving, and unified governance

Get Started

Snowflake users can reduce credit consumption, federate beyond Snowflake's ecosystem, and add AI analytics — all through Dremio Cloud. The combination of Reflections (offloading repetitive dashboard and report queries), federation (joining Snowflake with PostgreSQL, S3, MongoDB, and other sources without ETL), and AI capabilities (Agent, MCP Server, SQL Functions) makes Dremio a natural complement to any Snowflake deployment.

Start by connecting Snowflake to Dremio Cloud, creating Reflections on your most-queried views, and monitoring the reduction in Snowflake credit consumption. Most organizations see measurable savings within the first week as dashboard queries shift to Dremio's Reflection cache. The setup takes minutes and the ROI is immediate.

Try Dremio Cloud free for 30 days and connect your Snowflake warehouse.

Connect Google BigQuery to Dremio Cloud: Cross-Cloud Analytics Without Data Movement

Alex Merced — Sun, 01 Mar 2026 18:00:00 GMT

Google BigQuery is Google Cloud's serverless data warehouse. If your organization uses Google Cloud Platform, BigQuery is where your analytics data, marketing attribution, Google Analytics exports, and machine learning model outputs live. BigQuery is powerful within Google's ecosystem, but it creates challenges when your data spans multiple clouds or when costs grow with usage.

BigQuery's on-demand pricing charges per terabyte scanned. For organizations with large datasets queried frequently — especially by dashboards that refresh automatically — this can result in monthly bills that grow unpredictably. And connecting BigQuery data to non-Google tools and other cloud providers requires data exports, cross-cloud networking, or third-party ETL platforms.

Dremio Cloud connects to BigQuery and queries it alongside data from AWS, Azure, on-premises databases, and any other connected source. You get multi-cloud federation without data movement, AI-powered analytics, and cost optimization through Reflections.

Data gravity is a real challenge for BigQuery users. Once data lands in BigQuery, Google's ecosystem encourages keeping everything there — Looker for BI, Vertex AI for ML, Cloud Dataflow for processing. But most enterprises aren't all-Google. They have data in AWS RDS, Azure SQL, S3 data lakes, and on-premises systems. Moving all that data into BigQuery is expensive (ingestion costs, ongoing storage) and creates vendor lock-in. Dremio's federation approach queries each source in place, avoiding the data gravity trap while still giving you unified analytics across your entire data estate.

Why BigQuery Users Need Dremio

Control BigQuery Costs with Reflections

BigQuery's on-demand pricing charges per terabyte scanned, regardless of whether you've run the same query before. A dashboard that refreshes every 15 minutes, querying the same 500GB table, generates substantial costs. Dremio's Reflections solve this: after the first query execution, Dremio caches the results as a pre-computed materialization. Subsequent queries that match the Reflection pattern are served from cache — no BigQuery scan, no per-TB charge.

For organizations with heavy dashboard and reporting workloads, this can reduce BigQuery costs by 50-80% on those specific query patterns.

Multi-Cloud Analytics

Your Google Analytics data is in BigQuery, your application database is in PostgreSQL (running on AWS RDS), your product catalog is in SQL Server (on Azure), and your raw event logs are in Amazon S3. Without a federation layer, joining these datasets requires building ETL pipelines for each source-destination pair. Dremio eliminates this: connect all four as sources and write a single SQL query that joins across them.

Unified Governance Across Clouds

BigQuery has IAM policies and column-level security within Google Cloud. But those policies don't extend to your PostgreSQL database, S3 data lake, or Snowflake warehouse. Dremio's Fine-Grained Access Control (FGAC) applies consistent row-level security and column masking across BigQuery and every other connected source. One governance policy, everywhere.

The Semantic Layer for AI

Raw BigQuery tables have technical column names and fragmented schemas. Dremio lets you create views that consolidate and rename these into business-friendly structures, then attach wiki descriptions and labels. This semantic layer makes your BigQuery data queryable by AI tools — both Dremio's built-in AI Agent and external AI clients through the MCP Server.

Prerequisites

Google Cloud project ID — the GCP project containing your BigQuery datasets
Service Account JSON key — a GCP service account with the BigQuery Data Viewer role (or custom role with bigquery.tables.getData, bigquery.jobs.create permissions)
Network access — Dremio Cloud connects to Google Cloud APIs over HTTPS
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect BigQuery to Dremio Cloud

1. Add the BigQuery Source

In the Dremio console, click "+" in the left sidebar and select Google BigQuery.

2. Configure Connection Details

Name: A descriptive identifier (e.g., bigquery-marketing or gcp-analytics). This appears in SQL queries as the source prefix.
Project ID: Your Google Cloud project ID (e.g., my-company-analytics-123456).
Service Account Key: Upload or paste the JSON key file for your service account.

3. Configure Advanced Settings

Setting	Purpose
Caching Enabled	Cache BigQuery metadata locally for faster schema browsing
Billing Project	Specify which GCP project is billed for queries (important for cross-project access)
Connection Properties	Custom parameters for the BigQuery connection

4. Set Reflection and Metadata Refresh

Reflection Refresh: How often Dremio re-queries BigQuery to update cached Reflections. Balance between data freshness and BigQuery scan costs.
Metadata Refresh: How often Dremio checks for new datasets or schema changes.

5. Set Privileges and Save

Optionally restrict access, then click Save.

Query BigQuery Data from Dremio

-- Query BigQuery marketing data
SELECT
  campaign_name,
  SUM(clicks) AS total_clicks,
  SUM(conversions) AS total_conversions,
  ROUND(SUM(conversions) * 100.0 / NULLIF(SUM(clicks), 0), 2) AS conversion_rate
FROM "bigquery-marketing".analytics.campaign_metrics
WHERE date >= '2024-01-01' AND date < '2024-07-01'
GROUP BY campaign_name
ORDER BY total_conversions DESC;

Federate BigQuery with Other Clouds

Join BigQuery marketing data with AWS-hosted application data and Azure revenue:

SELECT
  bq.campaign_name,
  bq.total_clicks,
  bq.total_conversions,
  SUM(pg.order_total) AS attributed_revenue,
  ROUND(SUM(pg.order_total) / NULLIF(bq.total_conversions, 0), 2) AS revenue_per_conversion
FROM (
  SELECT campaign_name, user_id, SUM(clicks) AS total_clicks, SUM(conversions) AS total_conversions
  FROM "bigquery-marketing".analytics.campaign_clicks
  WHERE date >= '2024-01-01'
  GROUP BY campaign_name, user_id
) bq
JOIN "postgres-orders".public.orders pg
  ON bq.user_id = pg.customer_id
  AND pg.order_date >= '2024-01-01'
GROUP BY bq.campaign_name, bq.total_clicks, bq.total_conversions
ORDER BY attributed_revenue DESC;

Three clouds, one query, zero ETL.

Build a Semantic Layer

CREATE VIEW analytics.gold.campaign_performance AS
SELECT
  bq.campaign_name,
  SUM(bq.clicks) AS total_clicks,
  SUM(bq.conversions) AS total_conversions,
  ROUND(SUM(bq.conversions) * 100.0 / NULLIF(SUM(bq.clicks), 0), 2) AS conversion_rate_pct,
  SUM(bq.cost) AS total_ad_spend,
  CASE
    WHEN SUM(bq.conversions) * 100.0 / NULLIF(SUM(bq.clicks), 0) > 5 THEN 'High Performer'
    WHEN SUM(bq.conversions) * 100.0 / NULLIF(SUM(bq.clicks), 0) > 2 THEN 'Average'
    ELSE 'Underperforming'
  END AS campaign_grade
FROM "bigquery-marketing".analytics.campaign_metrics bq
GROUP BY bq.campaign_name;

Navigate to the Catalog, click Edit (pencil icon), and Generate Wiki and Generate Tags to create business context for AI features.

AI-Powered Analytics on BigQuery Data

Dremio AI Agent

The built-in AI Agent lets users ask questions in plain English: "Which campaigns had the highest conversion rate this quarter?" The Agent reads your wiki descriptions to understand what "conversion rate" and "high performer" mean, then generates accurate SQL.

Dremio MCP Server

The Dremio MCP Server connects Claude, ChatGPT, and other AI clients to your Dremio data. Setup:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs (e.g., https://claude.ai/api/mcp/auth_callback)
Connect using mcp.dremio.cloud/mcp/{project_id}

A marketing executive can ask Claude "Compare our Q1 campaign performance against Q2 using the BigQuery data" and get governed, accurate results — no SQL required.

AI SQL Functions

Use AI directly in queries against BigQuery data:

-- Classify campaign performance with AI
SELECT
  campaign_name,
  total_clicks,
  conversion_rate_pct,
  AI_CLASSIFY(
    'Based on these marketing metrics, recommend a budget action',
    'Campaign: ' || campaign_name || ', Clicks: ' || CAST(total_clicks AS VARCHAR) || ', Conversion Rate: ' || CAST(conversion_rate_pct AS VARCHAR) || '%',
    ARRAY['Increase Budget', 'Maintain Budget', 'Decrease Budget', 'Pause Campaign']
  ) AS budget_recommendation
FROM analytics.gold.campaign_performance;

-- Generate executive summaries
SELECT
  campaign_name,
  AI_GENERATE(
    'Write a brief performance summary for this marketing campaign',
    'Campaign: ' || campaign_name || ', Clicks: ' || CAST(total_clicks AS VARCHAR) || ', Conversions: ' || CAST(total_conversions AS VARCHAR) || ', Spend: $' || CAST(total_ad_spend AS VARCHAR)
  ) AS performance_summary
FROM analytics.gold.campaign_performance
WHERE campaign_grade = 'High Performer';

AI_CLASSIFY categorizes data with AI. AI_GENERATE produces narrative text. Both run inside your SQL query.

Accelerate with Reflections

For dashboard queries that run repeatedly against BigQuery:

Build a view over your BigQuery data
Navigate to the view in the Catalog and click the Reflections tab
Choose Raw Reflection (full cache) or Aggregation Reflection (pre-computed metrics)
Select columns and set the refresh interval
Click Save

Subsequent matching queries hit the Reflection instead of scanning BigQuery. This is particularly valuable for BigQuery's on-demand pricing, where every scan costs money. A dashboard with 10 widgets refreshing every 15 minutes would generate 960 BigQuery scans per day; with Reflections refreshing hourly, Dremio serves 936 of those from cache.

Governance Across BigQuery and Other Sources

Dremio's Fine-Grained Access Control (FGAC) provides governance that works across BigQuery and every other source:

Column masking: Mask ad spend or conversion data from specific roles. A content creator sees campaign impressions but not revenue.
Row-level filtering: Regional marketers see only campaigns in their territory.
Unified policies: Same rules apply to BigQuery, PostgreSQL, S3, and all connected sources.

These policies apply across SQL Runner, BI tools, AI Agent, MCP Server, and Arrow Flight clients.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Looker: Ideal for Google Cloud environments — connect via JDBC
Tableau: Dremio connector for direct Arrow Flight access
Power BI: Dremio ODBC driver or native connector
Python/Pandas: pyarrow.flight for programmatic data access
dbt: dbt-dremio adapter for transformations

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query BigQuery data from their IDE. Ask Copilot "Show me campaign conversion rates from BigQuery" and get SQL generated from your semantic layer.

When to Keep Data in BigQuery vs. Migrate

Keep in BigQuery: Data consumed by Google-native tools (Looker, Google Data Studio, Vertex AI), data pipelines managed by Cloud Dataflow or Dataproc, datasets with BigQuery ML models, data shared via BigQuery analytics hub.

Migrate to Iceberg: Historical archive data, datasets queried by non-Google tools, data that benefits from Iceberg's time travel and automated compaction, workloads where BigQuery per-TB costs exceed value. Migrated Iceberg tables get Dremio's automatic maintenance and Autonomous Reflections.

For data staying in BigQuery, create manual Reflections to eliminate per-TB scan costs for repeated queries.

BigQuery Cost Optimization with Dremio

BigQuery Pricing Models

Model	How It's Priced	Dremio's Impact
On-Demand	$6.25 per TB scanned	Reflections eliminate repeat scans — 50-80% cost reduction
Editions (Standard/Enterprise/Enterprise Plus)	Slot reservations (autoscaling)	Reflections reduce slot utilization, enabling lower commitments
Flat Rate	Fixed slot reservations	Reflections free up slots for other workloads

Google Analytics 4 (GA4) Integration

BigQuery is the default export destination for Google Analytics 4 data. GA4 exports create daily event tables (events_YYYYMMDD) with nested schemas. Dremio handles this pattern:

-- Query GA4 events from BigQuery through Dremio
SELECT
  event_name,
  COUNT(*) AS event_count,
  COUNT(DISTINCT user_pseudo_id) AS unique_users,
  DATE_TRUNC('day', CAST(event_timestamp AS TIMESTAMP)) AS event_day
FROM "bigquery-analytics".analytics_12345678.events_*
WHERE event_name IN ('page_view', 'purchase', 'add_to_cart')
GROUP BY 1, 4
ORDER BY event_day DESC, event_count DESC;

By creating Reflections on GA4 views, you can serve real-time marketing dashboards without accumulating BigQuery scan costs.

Multi-Cloud Analytics Strategy

For organizations with data across Google Cloud, AWS, and Azure:

BigQuery holds your Google-native data (GA4, Google Ads, Cloud Storage exports)
S3 holds your AWS data lake (application logs, IoT telemetry)
Azure Storage holds your Microsoft ecosystem data (Power Platform exports, Azure services)
PostgreSQL/MySQL hold operational application data

Dremio federates across all four clouds, applies unified governance, and serves all BI tools from a single connection. This eliminates the need for cross-cloud ETL pipelines.

Get Started

BigQuery users can break out of Google Cloud's walled garden, reduce per-TB scan costs with Reflections, and enable AI analytics across their entire data estate. Whether you're running a single BigQuery project or managing data across dozens of GCP projects alongside AWS and Azure resources, Dremio provides the federation layer that makes multi-cloud analytics practical.

The combination of Reflections (eliminating repetitive per-TB charges), federation (joining BigQuery with non-Google sources without ETL), and AI capabilities (Agent, MCP Server, SQL Functions) transforms BigQuery from an isolated Google Cloud analytics tool into a connected node in your broader data ecosystem. Your marketing team asks the AI Agent questions about campaign performance and gets accurate answers drawn from BigQuery data enriched with context from your semantic layer.

Try Dremio Cloud free for 30 days and connect your BigQuery projects.

Connect Amazon Redshift to Dremio Cloud: Extend Your Warehouse with Federation and AI Analytics

Alex Merced — Sun, 01 Mar 2026 17:00:00 GMT

Amazon Redshift is AWS's managed data warehouse, designed for petabyte-scale analytics. If your organization chose Redshift for analytical workloads, you've built data pipelines, ETL jobs, and dashboards around it. But as data ecosystems grow, Redshift's limitations become painfully clear: connecting data outside Redshift requires ETL or Redshift Spectrum (additional cost per TB scanned), sharing Redshift data with non-AWS tools means exporting to S3, and Redshift's concurrency limits constrain how many dashboards and users can query simultaneously.

Dremio Cloud connects to Redshift and queries it alongside every other data source in your organization. Instead of moving all your data into Redshift, or exporting Redshift data out, Dremio federates across sources and accelerates repeated queries with Reflections so your Redshift cluster handles less load.

Redshift's concurrency scaling feature helps handle burst query volumes, but it charges per-second of additional cluster time. By routing repeated dashboard queries through Dremio Reflections, you reduce the need for concurrency scaling entirely — cached results are served without any Redshift cluster involvement. This difference is particularly impactful for organizations running dozens of auto-refreshing dashboards.

Redshift Data Sharing vs. Dremio Federation

Redshift Data Sharing allows sharing data between Redshift clusters. But it only works within the Redshift ecosystem — you can't share Redshift data with Snowflake, BigQuery, or PostgreSQL through Data Sharing. Dremio's federation provides a broader solution: join Redshift data with any connected source. Data Sharing works for Redshift-to-Redshift use cases; Dremio handles everything else.

Redshift Serverless Consideration

With Redshift Serverless, you pay per RPU-second consumed. Every query, including repeated dashboard queries, consumes RPUs. Dremio Reflections eliminate RPU consumption for cached queries — a direct and measurable cost reduction. For Serverless users, the ROI from Reflections is immediately visible in the AWS billing dashboard.

Redshift's RA3 instances introduced compute-storage separation using Managed Storage backed by S3. While this improved scalability, all queries still consume RA3 compute resources. Dremio provides a complementary compute layer: Reflections handle repetitive analytical workloads while RA3 focuses on the data transformations and ingestion pipelines that require Redshift's native capabilities. This architectural separation — Redshift for data engineering, Dremio for analytics serving — maximizes the value of both platforms.

Why Redshift Users Need Dremio

Extend Redshift Without Spectrum Costs

Redshift Spectrum charges per TB scanned against S3. Dremio's federation queries S3 data directly through its own engine without per-TB charges. You still get SQL joins between Redshift and S3 data — Dremio handles the federation transparently.

Reduce Redshift Cluster Costs

Redshift pricing scales with cluster size (RA3, DC2, or Serverless credits). Analytical dashboards that run the same queries repeatedly consume cluster resources on every refresh. Dremio's Reflections serve cached results for matching queries, offloading that load from Redshift. For organizations with heavy dashboard workloads, this can reduce the Redshift cluster size needed.

Multi-Warehouse Federation

Your Redshift warehouse holds sales data, but your Snowflake instance has marketing data, your BigQuery project has Google Analytics data, and your PostgreSQL database has CRM data. Dremio federates across all four in a single query.

External Queries

Dremio supports external queries against Redshift, allowing you to run Redshift-native SQL (including Redshift-specific functions like APPROXIMATE COUNT(DISTINCT), window functions, and late binding views) through Dremio when needed.

AI Analytics

Dremio's semantic layer, AI Agent, MCP Server, and AI SQL Functions add natural language querying and AI enrichment to Redshift data without building a separate BI layer.

Prerequisites

Redshift cluster endpoint (hostname) — from the Redshift console
Port — default 5439
Database name — your Redshift database
Username and password — Redshift database user with SELECT permissions
Network access — Redshift cluster must be publicly accessible, or configure VPC peering with Dremio Cloud
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Redshift to Dremio Cloud

1. Add the Source

Click "+" in the Dremio console and select Amazon Redshift.

2. Configure Connection Details

Name: Descriptive identifier (e.g., redshift-warehouse or sales-analytics).
Host: Your Redshift cluster endpoint (e.g., mycluster.xxxx.us-east-1.redshift.amazonaws.com).
Port: Default 5439.
Database: Your Redshift database name.

3. Set Authentication

Master Credentials (username/password) or Secret Resource URL (AWS Secrets Manager).

4. Configure Advanced Options

Setting	Purpose	Default
Record fetch size	Rows per batch from Redshift	200
Maximum Idle Connections	Connection pool management	8
Connection Idle Time	Seconds before idle connections close	60
Encrypt Connection	Enable SSL/TLS	Off

5. Set Reflection and Metadata Refresh, then Save

Query Redshift Data

SELECT
  date_trunc('month', sale_date) AS month,
  product_category,
  SUM(revenue) AS monthly_revenue,
  COUNT(DISTINCT customer_id) AS unique_customers,
  ROUND(SUM(revenue) / COUNT(DISTINCT customer_id), 2) AS revenue_per_customer
FROM "redshift-warehouse".public.sales
WHERE sale_date >= '2024-01-01'
GROUP BY 1, 2
ORDER BY 1, monthly_revenue DESC;

External Queries

Run Redshift-native SQL through Dremio when you need Redshift-specific functions:

SELECT * FROM TABLE(
  "redshift-warehouse".EXTERNAL_QUERY(
    'SELECT TOP 100 querytxt, elapsed, starttime FROM stl_query WHERE starttime > GETDATE() - 7 ORDER BY elapsed DESC'
  )
);

Federate Redshift with Other Sources

-- Join Redshift sales with PostgreSQL CRM and S3 marketing data
SELECT
  c.customer_name,
  c.segment,
  SUM(s.revenue) AS total_revenue,
  COUNT(s.sale_id) AS total_sales,
  m.campaign_name,
  m.attribution_channel,
  ROUND(SUM(s.revenue) / NULLIF(m.campaign_spend, 0), 2) AS roas
FROM "postgres-crm".public.customers c
JOIN "redshift-warehouse".public.sales s ON c.customer_id = s.customer_id
LEFT JOIN "s3-marketing".attribution.customer_campaigns m ON c.customer_id = m.customer_id
WHERE s.sale_date >= '2024-01-01'
GROUP BY c.customer_name, c.segment, m.campaign_name, m.attribution_channel, m.campaign_spend
ORDER BY total_revenue DESC;

Build a Semantic Layer

CREATE VIEW analytics.gold.sales_performance AS
SELECT
  s.product_category,
  date_trunc('month', s.sale_date) AS month,
  SUM(s.revenue) AS revenue,
  COUNT(*) AS transactions,
  COUNT(DISTINCT s.customer_id) AS unique_buyers,
  ROUND(SUM(s.revenue) / COUNT(*), 2) AS avg_transaction_value,
  CASE
    WHEN SUM(s.revenue) > 500000 THEN 'Top Performer'
    WHEN SUM(s.revenue) > 100000 THEN 'Solid'
    ELSE 'Emerging'
  END AS performance_tier
FROM "redshift-warehouse".public.sales s
GROUP BY s.product_category, date_trunc('month', s.sale_date);

In the Catalog, click Edit → Generate Wiki and Generate Tags.

AI-Powered Analytics on Redshift Data

Dremio AI Agent

The AI Agent lets users ask "What were our top performing product categories last quarter?" and generates accurate SQL from your semantic layer. The wiki descriptions attached to views tell the Agent what "top performing" means in your data context.

Dremio MCP Server

Connect Claude or ChatGPT to your Redshift data through Dremio:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs for your AI client
Connect via mcp.dremio.cloud/mcp/{project_id}

A VP of Sales asks Claude "Compare our Q1 revenue per customer across product categories using the Redshift data" and gets a governed, accurate benchmark without SQL.

AI SQL Functions

-- Generate strategic recommendations from sales data
SELECT
  product_category,
  revenue,
  performance_tier,
  AI_GENERATE(
    'Write a strategic recommendation for this product category',
    'Category: ' || product_category || ', Revenue: $' || CAST(revenue AS VARCHAR) || ', Tier: ' || performance_tier || ', Avg Transaction: $' || CAST(avg_transaction_value AS VARCHAR)
  ) AS strategic_recommendation
FROM analytics.gold.sales_performance
WHERE month = DATE_TRUNC('month', CURRENT_DATE - INTERVAL '1' MONTH);

-- Classify product categories for budget allocation
SELECT
  product_category,
  AI_CLASSIFY(
    'Based on this sales performance, classify the marketing budget priority',
    'Revenue: $' || CAST(revenue AS VARCHAR) || ', Customers: ' || CAST(unique_buyers AS VARCHAR) || ', Avg Transaction: $' || CAST(avg_transaction_value AS VARCHAR),
    ARRAY['Increase Investment', 'Maintain Investment', 'Optimize Spend', 'Reduce Budget']
  ) AS budget_recommendation
FROM analytics.gold.sales_performance
WHERE month >= DATE_TRUNC('month', CURRENT_DATE - INTERVAL '3' MONTH);

Accelerate with Reflections

Create Reflections on Redshift views for dashboard acceleration:

Navigate to the view in the Catalog
Click the Reflections tab
Choose Raw Reflection (full dataset cache) or Aggregation Reflection (pre-computed SUM/COUNT/AVG)
Select columns and aggregations
Set the Refresh Interval — balance freshness against Redshift cluster load
Click Save

BI tools connected via Arrow Flight get sub-second responses from Reflections instead of waiting for Redshift cluster processing. A Tableau dashboard refreshing every 15 minutes generates zero Redshift cluster load after the Reflection is built.

Governance Across Redshift and Other Sources

Dremio's Fine-Grained Access Control (FGAC) adds governance capabilities that work uniformly across Redshift and every other connected source:

Column masking: Mask sensitive revenue data or PII from specific roles. A marketing analyst sees conversion rates but not individual customer records.
Row-level filtering: Restrict data visibility based on user roles. Regional managers see only their region's sales data.
Unified policies: The same governance applies to Redshift, PostgreSQL, S3, and all other sources — no per-database security configuration.

These policies apply across all access methods: SQL Runner, BI tools, AI Agent, MCP Server, and Arrow Flight clients.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC for BI tools:

Tableau: Use the Dremio connector for direct Arrow Flight access
Power BI: Use Dremio's ODBC driver or native connector
Python/Pandas: Use pyarrow.flight for programmatic data access
Looker: Connect via JDBC
dbt: Use dbt-dremio for transformation workflows

All queries benefit from Reflections, governance, and the semantic layer — whether the underlying data comes from Redshift or any other source.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration enables developers to query Redshift data from their IDE. Ask Copilot "Show me sales performance by category from Redshift" and it generates SQL using your semantic layer, eliminating context switching between tools.

When to Keep Data in Redshift vs. Migrate

Keep in Redshift: Data actively used by Redshift-native tools and materializations, workloads with existing Redshift-based ETL pipelines, datasets managed by Redshift's automatic table optimization (sort keys, distribution styles).

Migrate to Iceberg: Historical data and archives, datasets consumed primarily by non-Redshift tools, data where Redshift cluster costs exceed analytical value. Migrated Iceberg tables benefit from Dremio's automatic compaction, time travel, Autonomous Reflections, and zero per-query storage costs.

For data that stays in Redshift, create manual Reflections to reduce cluster load. For migrated Iceberg data, Dremio handles optimization automatically.

Redshift Cost Optimization with Dremio

Redshift Pricing Models

Model	How It's Priced	Dremio's Impact
RA3 Provisioned	Per-node-hour + managed storage	Reflections reduce node utilization, enabling cluster downsizing
DC2 Provisioned	Per-node-hour, SSD storage included	Same as RA3 — lower utilization means fewer nodes needed
Serverless	Per RPU-hour (compute consumed)	Reflections eliminate RPU consumption for cached queries
Spectrum	Per TB scanned in S3	Dremio queries S3 directly without per-TB charges

Quantifying Savings

A typical dashboard workload might include:

20 production dashboards, each refreshing every 15 minutes
50+ ad-hoc queries per day from analysts
Weekly scheduled reports generating 100+ queries

With Dremio Reflections, only the Reflection refresh queries hit Redshift. If Reflections refresh hourly:

Dashboard queries drop from 1,920/day to 24/day (hourly Reflection refresh × 24 hours) — a 98.7% reduction
Ad-hoc queries matching Reflection patterns are served from cache — zero Redshift load
Scheduled reports matching Reflections run instantly

Migration Strategy

Assess: Identify Redshift tables by query frequency and size. High-frequency, read-heavy tables are prime candidates for Reflections.
Accelerate: Create Reflections on the 10-20 most-queried views. Monitor Redshift cluster utilization.
Right-size: As utilization drops, reduce Redshift node count or switch from Provisioned to Serverless.
Migrate: Move historical and archival data from Redshift to Iceberg tables. Use CREATE TABLE ... AS SELECT in Dremio.
Optimize: Continue moving more tables as Redshift costs decrease and Dremio handles more workloads.

Get Started

Redshift users can extend their warehouse with federation, reduce cluster costs with Reflections, add AI analytics, and apply unified governance across their entire data estate. Whether you're running Redshift Provisioned, Serverless, or RA3, Dremio Reflections immediately reduce compute costs by caching repetitive queries. Start by connecting your cluster and creating Reflections on your most-queried views to see immediate results.

Try Dremio Cloud free for 30 days and connect your Redshift cluster.

Connect Azure Storage to Dremio Cloud: Query Your Microsoft Data Lake with SQL and AI

Alex Merced — Sun, 01 Mar 2026 16:00:00 GMT

Azure Storage is Microsoft's cloud storage platform, spanning Blob Storage, Azure Data Lake Storage Gen2 (ADLS Gen2), and Azure Files. If your organization uses Microsoft Azure, your data lake almost certainly lives in Azure Storage — Parquet files from Azure Data Factory pipelines, CSV exports from Azure SQL Database, JSON event streams from Azure Event Hubs, and raw data from Azure IoT Hub all land in Azure Storage containers.

Dremio Cloud connects directly to Azure Storage and lets you query these files in place using standard SQL. You don't need Azure Synapse Analytics (DWU-based pricing), Azure Databricks (DBU costs), or HDInsight (cluster management) to run analytical queries against your data lake. Dremio reads the data, accelerates repeated queries with Reflections, and federates Azure Storage with every other source in your data ecosystem.

Many Azure customers face a fragmented analytics experience: Synapse for warehouse workloads, Databricks for data engineering, Power BI for visualization, and Azure Data Explorer for log analytics — each with its own pricing model, access control, and query interface. Dremio consolidates the analytical layer by querying Azure Storage and other Azure (or non-Azure) services from a single SQL engine with unified governance and AI capabilities. Dremio reads Parquet, CSV, JSON, Delta Lake, and Apache Iceberg table formats from Azure Blob Storage and ADLS Gen2 containers. It pushes projection and filtering into its vectorized query engine and caches frequently accessed data on local NVMe drives (Columnar Cloud Cache, or C3) for near-instantaneous repeat queries.

Why Azure Storage Users Need Dremio

SQL Without Azure Synapse Costs

Azure Synapse serverless SQL charges per terabyte of data processed. For large datasets queried frequently — dashboard refreshes, ad-hoc exploration, scheduled reports — costs accumulate quickly. Dremio's Reflections eliminate repeat scans by caching pre-computed results. C3 caching further reduces Azure Storage API calls for frequently accessed files. Your first query scans Azure Storage; subsequent matching queries hit Dremio's cache.

Federation Beyond Azure

Your Azure data lake holds event data and ETL outputs, but your operational database is in PostgreSQL on AWS, your marketing data is in Google BigQuery, and your CRM is in Salesforce (exported to S3). Dremio federates across all three cloud providers in a single SQL query — no ADF (Azure Data Factory) pipelines needed.

Apache Iceberg Table Management

Create Iceberg tables backed by Azure Storage (or Dremio-managed storage) with full DML support (INSERT, UPDATE, DELETE, MERGE). Dremio automatically handles compaction, manifest rewriting, clustering, and vacuuming. No manual OPTIMIZE jobs, no maintenance scripts.

AI on Azure Data

Dremio's AI Agent, MCP Server, and AI SQL Functions make your Azure data queryable by non-technical users and external AI tools. Build a semantic layer over your Azure files, and let AI do the heavy lifting.

Prerequisites

Azure Storage Account with Blob Storage or ADLS Gen2 enabled
Authentication: Azure Active Directory OAuth 2.0, Shared Access Key, or Shared Access Signature (SAS) Token
Container names you want to query
Network access from Dremio Cloud to Azure Storage endpoints
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Azure Storage to Dremio Cloud

1. Add the Source

Click "+" in the Dremio console and select Azure Storage.

2. Configure Connection Details

Name: Descriptive identifier (e.g., azure-datalake or adls-analytics).
Account: Your Azure Storage account name.

3. Set Authentication

Choose from:

Azure AD (OAuth 2.0): Most secure, uses service principal or managed identity.
Shared Access Key: Full access to the storage account. Simpler but less granular.
SAS Token: Scoped, time-limited access.

4. Configure Advanced Options

Setting	Purpose	Default
Root Path	Starting container/path	`/` (all containers)
CTAS Format	Default CREATE TABLE format	Iceberg recommended
Encrypt Connection	Enable HTTPS	On
Enable partition column inference	Extract partition keys from folder structures	Off
Enable file status check	Verify file existence before reads	On

5. Set Reflection and Metadata Refresh, then Save

Query Azure Storage Data

-- Query Parquet files directly
SELECT transaction_id, customer_id, amount, transaction_date
FROM "azure-datalake".sales."transactions.parquet"
WHERE transaction_date >= '2024-01-01' AND amount > 100
ORDER BY amount DESC;

-- Query partitioned data (Hive-style partitions)
SELECT region, product_category, SUM(revenue) AS total_revenue
FROM "azure-datalake".sales.transactions
WHERE year = '2024' AND quarter = 'Q1'
GROUP BY region, product_category
ORDER BY total_revenue DESC;

Federate with Other Clouds

-- Join Azure data with AWS and Google Cloud sources
SELECT
  c.customer_name,
  c.segment,
  SUM(a.amount) AS azure_revenue,
  COUNT(s.event_id) AS aws_events,
  bq.campaign_clicks
FROM "postgres-crm".public.customers c
LEFT JOIN "azure-datalake".sales.transactions a ON c.customer_id = a.customer_id
LEFT JOIN "s3-events".analytics.user_events s ON c.customer_id = s.user_id
LEFT JOIN "bigquery-marketing".analytics.customer_clicks bq ON c.customer_id = bq.user_id
GROUP BY c.customer_name, c.segment, bq.campaign_clicks
ORDER BY azure_revenue DESC;

Four clouds, one query.

Build a Semantic Layer

CREATE VIEW analytics.gold.customer_transactions AS
SELECT
  a.customer_id,
  a.transaction_date,
  a.amount,
  CASE
    WHEN a.amount > 1000 THEN 'High Value'
    WHEN a.amount > 100 THEN 'Standard'
    ELSE 'Micro'
  END AS transaction_tier,
  DATE_TRUNC('month', a.transaction_date) AS transaction_month
FROM "azure-datalake".sales.transactions a
WHERE a.transaction_date >= '2024-01-01';

In the Catalog, click Edit (pencil icon) → Generate Wiki and Generate Tags.

AI-Powered Analytics on Azure Data

Dremio AI Agent

Ask questions in plain English: "What's our total revenue from high-value transactions this quarter?" The AI Agent reads your wiki descriptions and generates accurate SQL against your Azure data.

Dremio MCP Server

Connect Claude or ChatGPT to your Azure data:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs
Connect using mcp.dremio.cloud/mcp/{project_id}

An operations team member can ask Claude "Show me a summary of our Azure sales data by region this month" — no SQL required.

AI SQL Functions

-- Classify transactions with AI
SELECT
  transaction_id,
  amount,
  AI_CLASSIFY(
    'Based on this transaction, classify the likely purchase category',
    'Amount: $' || CAST(amount AS VARCHAR) || ', Date: ' || CAST(transaction_date AS VARCHAR),
    ARRAY['Subscription', 'One-Time Purchase', 'Refund', 'Upgrade']
  ) AS inferred_category
FROM "azure-datalake".sales.transactions
WHERE transaction_date = CURRENT_DATE;

-- Generate data quality summaries
SELECT
  transaction_month,
  COUNT(*) AS total_transactions,
  AI_GENERATE(
    'Write a one-sentence summary of this month data quality',
    'Transactions: ' || CAST(COUNT(*) AS VARCHAR) || ', Avg Amount: $' || CAST(ROUND(AVG(amount), 2) AS VARCHAR) || ', Nulls: ' || CAST(SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) AS VARCHAR)
  ) AS quality_summary
FROM analytics.gold.customer_transactions
GROUP BY transaction_month;

Create Iceberg Tables from Azure Data

Promote raw Azure files into managed Iceberg tables with full ACID transaction support:

CREATE TABLE analytics.bronze.azure_events AS
SELECT event_type, user_id, CAST(event_timestamp AS TIMESTAMP) AS event_time, payload
FROM "azure-datalake".events."raw_events.parquet"
WHERE event_type IS NOT NULL;

Iceberg tables benefit from automatic compaction, time travel, results caching, and Autonomous Reflections. You can also use time travel to query historical states:

-- Query as table existed 7 days ago
SELECT * FROM analytics.bronze.azure_events
AT TIMESTAMP '2024-06-01 00:00:00';

Governance on Azure Data

Dremio's Fine-Grained Access Control (FGAC) adds governance capabilities that Azure Storage doesn't provide natively:

Column masking: Mask PII fields (email, IP address, user ID) from specific roles. Marketing analysts see aggregated metrics but not individual user data.
Row-level filtering: Automatically filter data by the querying user's role. A regional manager sees only their region's Azure data.
Unified policies: Same governance applies whether data comes from Azure Storage, PostgreSQL, BigQuery, or any other connected source.

These policies apply across SQL Runner, BI tools (via Arrow Flight/ODBC), AI Agent queries, and MCP Server interactions.

Connect BI Tools via Arrow Flight

Dremio's Arrow Flight connector provides 10-100x faster data transfer than JDBC/ODBC. After building views over your Azure data:

Power BI: Use Dremio's native connector or ODBC driver — ideal for Azure-centric organizations
Tableau: Use the Dremio connector for direct Arrow Flight access
Python/Pandas: Use pyarrow.flight for high-speed data access
dbt: Use dbt-dremio adapter for SQL-based transformations
Looker: Connect via JDBC

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Azure data directly from their IDE. Ask Copilot "Show me daily transaction trends from Azure storage" and it generates SQL using your semantic layer — without leaving your development environment.

Reflections and C3 Caching

For frequently queried Azure Storage data, create Reflections to pre-compute results:

Navigate to the view in the Catalog
Click the Reflections tab and create a Raw or Aggregation Reflection
Select columns and set the refresh interval

C3 (Columnar Cloud Cache) automatically caches frequently accessed file data on local NVMe drives for sub-second access. You don't configure C3 manually — it works transparently.

When to Keep Data in Azure Storage vs. Migrate to Iceberg

Keep as raw files: Data landing zones for Azure Data Factory, files consumed by Azure-native services (Databricks, Synapse, Azure ML), raw data in formats required by other tools.

Migrate to Iceberg tables: Analytical datasets consumed by SQL queries, data that benefits from ACID transactions and time travel, historical data needing snapshot management, datasets consumed by BI tools and AI agents.

For raw Azure files, query through the connector and create manual Reflections. For Iceberg tables (either in Dremio's Open Catalog or external catalogs), Dremio provides automated compaction, Autonomous Reflections, and zero-maintenance performance optimization.

Azure Storage Tiers and Dremio Performance

Azure Storage offers multiple access tiers that affect query performance:

Tier	Access Latency	Cost	Dremio Recommendation
Hot	Milliseconds	Highest storage, lowest access	Active analytics data — best performance
Cool	Milliseconds	Lower storage, higher access	Infrequent queries — still fast
Cold	Milliseconds	Even lower storage, higher access	Archival analytics — acceptable latency
Archive	Hours (rehydrate required)	Lowest storage, highest access	Not suitable for Dremio queries — rehydrate first

For optimal Dremio performance, keep analytical data in Hot or Cool tiers. Use Azure lifecycle management policies to automatically transition data between tiers based on last access time.

ADLS Gen2 vs. Blob Storage

Dremio's Azure Storage connector supports both Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Blob Storage:

ADLS Gen2 is the recommended option for analytical workloads:

Hierarchical namespace enables true directory operations (faster metadata operations)
Fine-grained POSIX-like permissions for directory and file-level access
Optimized for large-scale analytics workloads
Required for Iceberg table creation and Azure Synapse integration

Azure Blob Storage works for read-only access to existing file datasets but lacks hierarchical namespace features.

When creating your Azure Storage source in Dremio, specify the storage account and container. For ADLS Gen2 accounts, Dremio automatically uses the abfss:// protocol for optimized access.

Azure-Specific Integration Patterns

Azure Data Factory + Dremio

Azure Data Factory (ADF) lands data into Azure Storage containers. Dremio queries this data in place:

ADF pipelines extract from Azure SQL, Cosmos DB, or external APIs
ADF writes Parquet files to ADLS Gen2 containers
Dremio queries the Parquet files via the Azure Storage connector
Dremio creates Iceberg tables from the Parquet data for optimized analytics

Azure Synapse + Dremio + Azure Storage

Connect both Azure Synapse and Azure Storage to Dremio Cloud. Dremio federates data across both:

Synapse contains summarized, modeled data
Azure Storage contains raw files and Iceberg tables
Dremio joins both sources in a single query

This eliminates the need to load all Azure Storage data into Synapse, reducing Synapse DWU consumption and costs.

Get Started

Azure Storage users can query their cloud data lake with SQL, federate with other sources, build a semantic layer, and enable AI analytics — all without data movement or ETL pipelines.

Try Dremio Cloud free for 30 days and connect your Azure Storage accounts alongside your other data sources.

Connect Amazon S3 to Dremio Cloud: Query Your Data Lake with SQL, Federation, and AI

Alex Merced — Sun, 01 Mar 2026 15:00:00 GMT

Amazon S3 is the default landing zone for data in the cloud. Log files, Parquet datasets, CSV exports, JSON events, IoT telemetry, and raw data dumps — it all ends up in S3 buckets. But S3 is storage, not an analytics engine. You can't run SQL against S3 natively. To query it, you need Amazon Athena (per-TB pricing), AWS Glue ETL jobs (cluster management), or a data warehouse that imports the data. All add cost, complexity, and latency.

Dremio Cloud connects directly to S3 and lets you query files in place using standard SQL. Dremio reads Parquet, CSV, JSON, Delta Lake, and Apache Iceberg table formats. It pushes projection and filter operations into its vectorized query engine and caches frequently accessed data on local NVMe drives (Columnar Cloud Cache, or C3) for near-instantaneous repeat queries.

For organizations with hundreds or thousands of S3 buckets accumulated over years, data lake sprawl is a major challenge. Data lands in S3 from application logs, CDC pipelines, third-party integrations, and manual uploads — often without consistent naming conventions, schemas, or documentation. Dremio provides the organizational layer: connect S3 buckets, create views that standardize column names and types, build a semantic layer with wiki descriptions, and expose clean datasets to analysts and AI tools. This turns an unstructured "data swamp" into a governed, queryable data lake.

Why S3 Users Need Dremio

SQL on Your Data Lake Without Athena Costs

Athena charges per terabyte of data scanned. For large datasets queried frequently — dashboards refreshing every 15 minutes, analysts exploring data, scheduled reports — costs grow unpredictably. Dremio's Reflections pre-compute results so repeated queries don't re-scan S3. C3 caching further reduces S3 GET requests. You pay for Dremio compute time, not per-TB scanned.

Format Flexibility

Dremio reads Parquet, CSV, JSON, Avro, Delta Lake, and Apache Iceberg from S3. You don't need to convert everything to one format before querying. Mixed-format data lakes work out of the box.

Federation with Databases and Warehouses

Your event data is in S3, but your customer data is in PostgreSQL, your financial data is in Snowflake, and your marketing data is in BigQuery. Dremio joins across all of them in a single SQL query without copying data between systems.

Apache Iceberg Table Management

Create Iceberg tables in Dremio's Open Catalog (backed by S3 or Dremio-managed storage) with full DML support. Dremio automatically handles compaction (merging small files), manifest rewriting, clustering, and vacuuming — no manual OPTIMIZE jobs needed.

AI on S3 Data

Dremio's AI Agent, MCP Server, and AI SQL Functions make your raw S3 files queryable by business users and external AI tools — no data engineering required.

Prerequisites

AWS Account with S3 access
IAM Role or Access Key/Secret Key with s3:GetObject, s3:ListBucket, and s3:GetBucketLocation permissions
Bucket names or specific paths you want to query
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect S3 to Dremio Cloud

1. Add the Source

Click "+" and select Amazon S3.

2. Configure Connection

Name: Descriptive identifier (e.g., s3-datalake or event-logs).
Authentication: IAM Role ARN (recommended) or Access Key/Secret Key.

3. Configure Advanced Options

Setting	Purpose	When to Use
Root Path	Starting path in the bucket	Restrict to subfolder: `/data/analytics/`
Allowlisted Buckets	Limit which buckets appear	Multi-bucket accounts
Enable partition column inference	Extract partition keys from folders	Hive-style partitioned data
Default CTAS Format	CREATE TABLE format	Iceberg recommended
Encrypt Connection	Enable SSL	Always recommended
Requester Pays	For requester-pays buckets	Cross-account access
Enable compatibility mode	S3-compatible storage	MinIO, R2, etc.
Connection Properties	Custom settings	`fs.s3a.endpoint` for non-AWS

4. Set Reflection and Metadata Refresh, then Save

Query S3 Data

-- Query Parquet files
SELECT event_type, user_id, event_timestamp, page_url
FROM "s3-datalake".events."user_events.parquet"
WHERE event_type = 'purchase' AND event_timestamp > '2024-01-01';

-- Query partitioned data (e.g., year=2024/month=01/)
SELECT region, product_category, SUM(revenue) AS total_revenue
FROM "s3-datalake".sales.transactions
GROUP BY region, product_category
ORDER BY total_revenue DESC;

Federate S3 with Other Sources

SELECT
  c.customer_name,
  c.segment,
  COUNT(e.event_id) AS s3_events,
  SUM(CASE WHEN e.event_type = 'purchase' THEN e.revenue ELSE 0 END) AS s3_revenue,
  pg.lifetime_value AS crm_lifetime_value
FROM "postgres-crm".public.customers c
LEFT JOIN "s3-datalake".events.user_events e ON c.customer_id = e.user_id
LEFT JOIN "postgres-crm".public.customer_metrics pg ON c.customer_id = pg.customer_id
GROUP BY c.customer_name, c.segment, pg.lifetime_value
ORDER BY s3_revenue DESC;

Create Iceberg Tables from S3 Data

Promote raw S3 files into managed Iceberg tables:

CREATE TABLE analytics.bronze.clean_events AS
SELECT event_type, user_id, CAST(event_timestamp AS TIMESTAMP) AS event_time, page_url, revenue
FROM "s3-datalake".events."user_events.parquet"
WHERE event_type IS NOT NULL;

The Iceberg table benefits from automatic compaction, time travel, results caching, and Autonomous Reflections.

S3-Compatible Storage

Dremio's S3 connector works with S3-compatible storage:

MinIO: Enable compatibility mode, set fs.s3a.endpoint to your MinIO endpoint.
Cloudflare R2: Same pattern, with R2's S3-compatible endpoint.
DigitalOcean Spaces: Compatibility mode + custom endpoint.
Amazon FSx for NetApp ONTAP: Set the S3 Access Point alias as the root path, ensure IAM permissions include FSx-specific actions.

Build a Semantic Layer

CREATE VIEW analytics.gold.event_metrics AS
SELECT
  DATE_TRUNC('day', CAST(event_timestamp AS TIMESTAMP)) AS event_date,
  event_type,
  COUNT(*) AS event_count,
  COUNT(DISTINCT user_id) AS unique_users,
  SUM(revenue) AS daily_revenue,
  CASE
    WHEN COUNT(*) > 10000 THEN 'High Activity'
    WHEN COUNT(*) > 1000 THEN 'Normal Activity'
    ELSE 'Low Activity'
  END AS activity_level
FROM "s3-datalake".events.user_events
GROUP BY 1, 2;

In the Catalog, click Edit → Generate Wiki and Generate Tags.

AI-Powered Analytics on S3 Data

Dremio AI Agent

Ask "What's our daily purchase revenue trend this month?" and the AI Agent generates SQL from your semantic layer. The wiki descriptions guide the Agent's understanding of event types and metrics.

Dremio MCP Server

Connect Claude or ChatGPT to your S3 data:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs
Connect via mcp.dremio.cloud/mcp/{project_id}

A product analyst asks Claude "Analyze user engagement patterns from S3 event data this week" and gets governed results.

AI SQL Functions

-- Classify events with AI
SELECT
  event_type,
  event_count,
  AI_CLASSIFY(
    'Based on this event pattern, classify the business impact',
    'Event: ' || event_type || ', Count: ' || CAST(event_count AS VARCHAR) || ', Revenue: $' || CAST(daily_revenue AS VARCHAR),
    ARRAY['Revenue Driver', 'Engagement Signal', 'Support Indicator', 'Churn Signal']
  ) AS business_impact
FROM analytics.gold.event_metrics
WHERE event_date = CURRENT_DATE - INTERVAL '1' DAY;

-- Process unstructured data from S3
SELECT
  file['path'] AS file_path,
  AI_GENERATE(
    'Extract key information from this document',
    ('Summarize the main topics in this file', file)
    WITH SCHEMA ROW(summary VARCHAR, category VARCHAR)
  ) AS extracted_info
FROM TABLE(LIST_FILES('@"s3-datalake"/documents/'))
WHERE file['path'] LIKE '%.pdf';

AI_GENERATE with file references can process unstructured documents (PDFs, images) stored in S3 directly in SQL queries.

Reflections and C3 Caching

For frequently queried S3 data, Dremio provides two layers of acceleration:

Reflections pre-compute query results. Create them on your semantic layer views:

In the Catalog, navigate to the view
Click the Reflections tab
Choose Raw or Aggregation Reflections
Select columns and set the refresh interval
Click Save

C3 (Columnar Cloud Cache) automatically caches frequently accessed file data on local NVMe drives. C3 works transparently — no configuration needed. When Dremio reads S3 files, it caches the columnar data locally. Subsequent reads of the same files come from NVMe instead of S3, eliminating S3 GET request costs and latency.

Together, Reflections and C3 mean that frequently executed queries against S3 data run in milliseconds, not seconds.

Governance on S3 Data

S3 has bucket-level IAM policies, but no column-level masking or row-level filtering. Dremio's Fine-Grained Access Control (FGAC) adds these capabilities:

Column masking: Mask PII fields (email, IP, user ID) from specific roles. Data engineers see everything; marketing analysts see aggregated metrics only.
Row-level filtering: Restrict data by user role. Regional analysts see only their region's events.
Unified policies: Same governance applies across S3, PostgreSQL, Snowflake, and all other sources.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Tableau: Dremio connector for direct Arrow Flight access
Power BI: Dremio ODBC driver or native connector
Python/Pandas: pyarrow.flight for programmatic access
Looker: Connect via JDBC
dbt: dbt-dremio adapter for transformation workflows

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot lets developers query S3 data from their IDE. Ask Copilot "Show me purchase event trends from S3 data this week" and get SQL generated using your semantic layer.

S3 Data Organization Best Practices

How you organize data in S3 directly impacts Dremio's query performance:

Partition Strategy

Hive-style partitions (year=2024/month=01/day=15/) enable Dremio to skip irrelevant partitions during query planning. The right partition key depends on your query patterns:

Time-based queries: Partition by year/month/day or year/month. Dremio reads only the partitions matching your WHERE clause.
Regional queries: Partition by region/date for multi-region datasets.
Mixed access: Partition by the most common filter column first (e.g., region/year/month).

Avoid over-partitioning (too many small files per partition) or under-partitioning (too few partitions with huge files). Aim for partition sizes between 128 MB and 1 GB.

File Format Recommendations

Format	Best For	Dremio Support
Parquet	Structured analytics data	Full support, columnar optimization
Apache Iceberg	ACID transactions, time travel	Full read/write support
Delta Lake	Databricks ecosystem compatibility	Read support
JSON	Semi-structured event data	Full support, schema inference
CSV	Legacy data imports	Full support, limited performance
Avro	Schema-evolved event streams	Read support

For analytical workloads, convert CSV and JSON files to Parquet or Iceberg for 10-50x better query performance. Dremio can perform this conversion:

CREATE TABLE analytics.bronze.events_optimized AS
SELECT * FROM "s3-datalake".raw."events.csv";

This creates an Iceberg table from CSV data, giving you columnar storage, automatic compaction, and time travel.

Data Lake Layers

Organize your S3 bucket with a medallion architecture:

raw/ — Landing zone for incoming data (CSV, JSON, Parquet from external sources)
bronze/ — Cleaned, typed versions of raw data (Iceberg tables)
silver/ — Joined, deduplicated, enriched datasets
gold/ — Business-ready views and aggregations for the semantic layer

Dremio's SQL engine handles the transformations between layers using CREATE TABLE AS SELECT and MERGE statements — no external ETL tools needed.

When to Use S3 vs. Other Storage

Use S3 when: Your data originates in AWS, you need cost-effective long-term storage, you want to use Apache Iceberg tables, your data is in file formats (Parquet, JSON, CSV).

Use managed databases when: Your data requires real-time OLTP operations, your applications need row-level transactions, your data model is heavily relational.

Dremio federates across both — S3 for your data lake and databases for operational data, in a single query.

Get Started

Amazon S3 is the most common data lake storage layer. Dremio Cloud turns it into a queryable, federated, AI-ready analytics platform without Athena costs or data warehouse ETL. Whether your S3 data is in Parquet, CSV, JSON, or Iceberg format, Dremio reads it directly and makes it available for SQL queries, cross-source joins, and AI-powered analytics.

Start by connecting your primary S3 bucket to Dremio Cloud. Create views that standardize your data into business-friendly structures, add wiki descriptions for the AI Agent, and build Reflections on frequently accessed datasets. Within hours, your S3 data lake transforms from raw file storage into a governed, AI-ready analytical platform. No infrastructure to manage and no data to move.

Try Dremio Cloud free for 30 days and connect your S3 buckets.

Connect SAP HANA to Dremio Cloud: Unlock Analytics Beyond the SAP Ecosystem

Alex Merced — Sun, 01 Mar 2026 14:00:00 GMT

SAP HANA is the in-memory database platform that powers SAP S/4HANA, SAP BW/4HANA, and custom enterprise applications across finance, manufacturing, logistics, and supply chain. It's fast for SAP-native analytics — real-time financial reporting, material requirements planning, and production analytics run directly on HANA's in-memory columnar engine. But SAP HANA exists in a walled garden.

Connecting HANA data to non-SAP tools requires SAP Data Intelligence, SAP Business Technology Platform (BTP), or custom ABAP extractors — all of which add significant cost and complexity. Sharing HANA data with teams that don't use SAP tools (marketing running Tableau, data science using Python, operations using Power BI) means building export pipelines that duplicate data, add latency, and create governance gaps.

Dremio Cloud connects directly to SAP HANA and queries it alongside your other data sources with standard SQL. No SAP-specific middleware. No data extraction. Your HANA data stays in place and joins with S3, PostgreSQL, BigQuery, Snowflake, or any other connected source in a single SQL query.

Why SAP HANA Users Need Dremio

Break Out of the SAP Ecosystem

SAP Analytics Cloud and SAP BusinessObjects work well with HANA, but connecting HANA data to Tableau, Power BI, Looker, or Python-based analytics requires additional middleware, gateway servers, or data export. Dremio provides a vendor-neutral SQL layer that connects HANA to any BI tool via Arrow Flight (high-performance columnar data transfer) or standard ODBC/JDBC.

Cross-Platform Analytics

Your SAP data covers finance (GL accounts, AP/AR, cost centers) and supply chain (material masters, purchase orders, production orders). But your CRM data is in Salesforce (exported to S3), your support ticket data is in PostgreSQL, and your marketing attribution data is in Google BigQuery. Without a federation layer, combining these with SAP data requires building custom pipelines for each source. Dremio federates across all sources in a single query.

Reduce HANA Memory Pressure

SAP HANA licenses are tied to memory allocation — the more memory provisioned, the higher the license cost. Running analytical workloads in HANA consumes memory resources that compete with transactional OLTP operations. Dremio's Reflections offload repeated analytical queries from HANA's engine, reducing memory pressure and potentially allowing you to right-size your HANA memory allocation.

AI Analytics on SAP Data

SAP's AI capabilities (SAP Joule, embedded analytics) are tightly coupled to SAP applications. Dremio's AI Agent, MCP Server, and AI SQL Functions provide AI analytics that span SAP and non-SAP data sources — enabling cross-functional insights that SAP's tools can't deliver alone.

Prerequisites

SAP HANA hostname or IP address — the HANA server
Port number — typically 30015 for single-tenant, or 3XX15 for multi-tenant (XX = instance number)
Database name (required for multi-tenant HANA systems)
Username and password — HANA user with SELECT privileges on the schemas and tables you want to query
Network access — HANA port must be reachable from Dremio Cloud
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect SAP HANA to Dremio Cloud

1. Add the Source

Click "+" in the Dremio console and select SAP HANA.

2. Configure Connection Details

Name: Descriptive identifier (e.g., sap-hana or erp-analytics).
Host: HANA server hostname or IP.
Port: Default 30015 for single-tenant.
Database: Required for multi-tenant HANA systems.

3. Set Authentication

Master Credentials (username/password) or Secret Resource URL (AWS Secrets Manager).

4. Configure Advanced Options

Setting	Purpose	Default
Record fetch size	Rows per batch from HANA	200
Maximum Idle Connections	Connection pool management	8
Connection Idle Time	Seconds before idle connections close	60
Enable SSL	Encrypt the connection	Off
Connection Properties	Custom JDBC parameters	None

5. Set Reflection and Metadata Refresh, then Save

Query SAP HANA Data

-- Query material inventory data
SELECT material_id, material_desc, plant, stock_quantity, unit_of_measure
FROM "sap-hana".SAPABAP1.MARD
WHERE plant = '1000' AND stock_quantity > 100
ORDER BY stock_quantity DESC;

-- Financial reporting: GL Account balances
SELECT
  gl_account,
  company_code,
  fiscal_year,
  SUM(debit_amount) AS total_debits,
  SUM(credit_amount) AS total_credits,
  SUM(debit_amount) - SUM(credit_amount) AS net_balance
FROM "sap-hana".SAPABAP1.BSEG
WHERE fiscal_year = '2024'
GROUP BY gl_account, company_code, fiscal_year
ORDER BY net_balance DESC;

Federate SAP with Non-SAP Sources

-- Join SAP material data with external supplier and demand data
SELECT
  m.material_desc,
  m.stock_quantity,
  m.plant,
  s.supplier_name,
  s.lead_time_days,
  s.unit_cost,
  d.forecasted_demand_30d,
  CASE
    WHEN m.stock_quantity < d.forecasted_demand_30d * 0.5 THEN 'Critical - Reorder Now'
    WHEN m.stock_quantity < d.forecasted_demand_30d THEN 'Watch - Order Soon'
    ELSE 'Adequate'
  END AS inventory_status
FROM "sap-hana".SAPABAP1.MARD m
JOIN "postgres-procurement".public.suppliers s ON m.material_id = s.material_id
LEFT JOIN "s3-forecasting".demand.material_forecasts d ON m.material_id = d.material_id AND m.plant = d.plant
WHERE s.lead_time_days < 14
ORDER BY s.unit_cost ASC;

SAP handles material masters, PostgreSQL has supplier details, S3 has demand forecasts — Dremio joins them all.

Build a Semantic Layer

CREATE VIEW analytics.gold.inventory_health AS
SELECT
  m.material_id,
  m.material_desc,
  m.plant,
  m.stock_quantity,
  CASE
    WHEN m.stock_quantity = 0 THEN 'Out of Stock'
    WHEN m.stock_quantity < 50 THEN 'Low Stock'
    WHEN m.stock_quantity < 200 THEN 'Adequate'
    ELSE 'Overstocked'
  END AS stock_status,
  ROUND(m.stock_quantity * s.unit_cost, 2) AS inventory_value_usd
FROM "sap-hana".SAPABAP1.MARD m
LEFT JOIN "postgres-procurement".public.suppliers s ON m.material_id = s.material_id;

In the Catalog, click Edit → Generate Wiki and Generate Tags. Create descriptions like "inventory_health: One row per material-plant combination showing current stock levels, status classification, and estimated inventory value in USD."

AI-Powered Analytics on SAP Data

Dremio AI Agent

The AI Agent lets users ask questions about SAP data in plain English: "Which materials are low stock at plant 1000?" or "What's the total inventory value across all plants?" The Agent reads your wiki descriptions, understands SAP terminology through the semantic layer, and generates accurate SQL.

This is transformative for SAP environments where only specialists know the table structures (MARD, BSEG, VBRK) and field names. The semantic layer translates SAP's technical schema into business language.

Dremio MCP Server

The Dremio MCP Server connects Claude and ChatGPT to your SAP data:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs (e.g., https://claude.ai/api/mcp/auth_callback)
Connect using mcp.dremio.cloud/mcp/{project_id}

A supply chain manager asks Claude "Show me all critical reorder items combining SAP inventory with supplier lead times" and gets actionable results without knowing SAP table names.

AI SQL Functions

-- Classify inventory risk with AI
SELECT
  material_desc,
  stock_quantity,
  stock_status,
  AI_CLASSIFY(
    'Based on inventory levels and value, recommend a procurement action',
    'Material: ' || material_desc || ', Stock: ' || CAST(stock_quantity AS VARCHAR) || ', Status: ' || stock_status || ', Value: $' || CAST(inventory_value_usd AS VARCHAR),
    ARRAY['Rush Order', 'Standard Reorder', 'Monitor', 'Liquidate Excess']
  ) AS procurement_action
FROM analytics.gold.inventory_health
WHERE stock_status IN ('Out of Stock', 'Low Stock', 'Overstocked');

-- Generate supplier evaluation summaries
SELECT
  s.supplier_name,
  AI_GENERATE(
    'Write a one-sentence supplier performance summary',
    'Supplier: ' || s.supplier_name || ', Lead Time: ' || CAST(s.lead_time_days AS VARCHAR) || ' days, Unit Cost: $' || CAST(s.unit_cost AS VARCHAR) || ', Materials Supplied: ' || CAST(COUNT(m.material_id) AS VARCHAR)
  ) AS performance_summary
FROM "postgres-procurement".public.suppliers s
JOIN "sap-hana".SAPABAP1.MARD m ON s.material_id = m.material_id
GROUP BY s.supplier_name, s.lead_time_days, s.unit_cost;

Reflections for SAP Analytics

SAP HANA is expensive to query for analytical workloads. Create Reflections on your semantic layer views to cache results:

Navigate to the view in the Catalog
Click the Reflections tab
Choose Raw Reflection (full cache) or Aggregation Reflection (pre-computed metrics)
Select columns and aggregations
Set the Refresh Interval — for SAP data that changes throughout the day, hourly is typical; for period-end data, daily or weekly
Click Save

Dashboard queries from Tableau or Power BI hit the Reflection instead of HANA, reducing memory consumption and license pressure. A financial reporting dashboard that queries HANA 96 times per day (15-minute refresh) with a Reflection refreshing every 2 hours consumes HANA resources only 12 times per day — an 87.5% reduction.

Governance on SAP Data

Dremio's Fine-Grained Access Control (FGAC) adds governance that SAP's built-in security doesn't extend to non-SAP tools:

Column masking: Mask salary data, cost center details, or GL account balances from specific roles. A supply chain analyst sees material inventory but not financial data.
Row-level filtering: Restrict data by company code, plant, or region based on the querying user's role. A plant manager sees only their plant's data.
Unified policies: Same governance applies across SAP HANA, PostgreSQL, S3, BigQuery, and all other sources.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server — ensuring consistent access control regardless of how data is queried.

Connect BI Tools via Arrow Flight

Dremio's Arrow Flight connector provides 10-100x faster data transfer than JDBC/ODBC. For SAP data, this eliminates the need for SAP BusinessObjects or SAP Analytics Cloud:

Tableau: Dremio connector — replaces SAP-specific Tableau drivers
Power BI: Dremio ODBC or native connector — no SAP Gateway needed
Python/Pandas: pyarrow.flight for data science on SAP data
Looker: Connect via JDBC
dbt: dbt-dremio for SQL-based transformations on SAP data

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query SAP data from their IDE. Ask Copilot "Show me low stock materials at plant 1000 from SAP" and get SQL generated using your semantic layer — without knowing SAP table names like MARD or BSEG.

When to Keep Data in HANA vs. Migrate

Keep in HANA: Transactional data actively used by SAP applications (OLTP), data with SAP-specific processing (ABAP reports, CDS views, BW extractors), master data referenced by SAP transactions, data subject to SAP transport management.

Migrate to Iceberg: Historical analytical data (closed fiscal periods, prior-year orders), datasets consumed by non-SAP tools, data where HANA memory cost exceeds analytical value, data needed for AI/ML workloads outside of SAP, archived transaction data that rarely changes.

For data staying in HANA, create manual Reflections to offload analytical queries. For migrated Iceberg data, Dremio provides automatic compaction, time travel, Autonomous Reflections, and zero per-query license costs.

SAP Landscape Integration

SAP HANA rarely exists in isolation. Dremio helps connect the SAP landscape with non-SAP analytics:

SAP S/4HANA Integration

S/4HANA stores business-critical data in HANA tables. Dremio connects to the underlying HANA database and reads these tables directly, bypassing the need for SAP BTP, SAP Analytics Cloud, or custom OData/RFC extractors. This gives analysts SQL access to S/4HANA data — sales orders, material documents, financial postings — alongside non-SAP sources.

SAP BW/4HANA Bridge

SAP BW/4HANA creates InfoProviders and ADSO tables in HANA. Dremio can query these underlying HANA tables, exposing BW-managed data to non-SAP BI tools. This is valuable for organizations consolidating from SAP Analytics Cloud and BW to a unified BI strategy.

Common SAP + Non-SAP Analytics Patterns

SAP Data (HANA)	Non-SAP Data	Analytics Use Case
Sales orders (VBAK/VBAP)	CRM opportunities (PostgreSQL)	Pipeline-to-revenue tracking
Material documents (MSEG)	IoT sensor data (S3)	Predictive maintenance
Financial postings (BSEG)	External market data (BigQuery)	Financial benchmarking
Employee master (PA0001)	Recruitment data (MongoDB)	Workforce analytics

Dremio's federation engine joins SAP tables with non-SAP sources without extracting SAP data to external systems — maintaining SAP as the system of record.

SAP HANA Licensing Considerations

SAP HANA licensing is based on memory allocation (RAM). Every analytical query consumes HANA memory resources. Dremio's Reflections offload analytical workloads from HANA, potentially allowing organizations to reduce HANA memory allocations and associated licensing costs.

Get Started

SAP HANA users can extend their SAP analytics beyond the SAP ecosystem — connect HANA, join it with every other source, and enable AI-driven analytics without SAP-specific middleware or additional SAP licenses. Start with Reflections to offload analytical queries from HANA's in-memory engine, then build a semantic layer for AI Agent access.

Try Dremio Cloud free for 30 days and connect your SAP HANA databases.

Connect IBM Db2 to Dremio Cloud: Modernize Mainframe Analytics with Federation and AI

Alex Merced — Sun, 01 Mar 2026 13:00:00 GMT

IBM Db2 is the relational database that powers critical applications across banking, insurance, government, healthcare, and manufacturing. For organizations running Db2 — particularly on IBM Z (mainframes) or IBM i — the database holds decades of transactional data: account balances, policy records, claim histories, manufacturing workflows, and government records. This data is enormously valuable for analytics but notoriously difficult to access outside the Db2/IBM ecosystem.

Traditional approaches to Db2 analytics involve CDC tools (IBM InfoSphere DataStage, Attunity), batch exports, or data replication to a separate analytics warehouse. These approaches are expensive, complex, and create stale copies of data that diverge from the source of truth.

Dremio Cloud connects directly to Db2 (Linux, UNIX, and Windows editions) and queries it alongside modern cloud sources in real time. No CDC infrastructure. No batch exports. Your Db2 data stays in place and joins with S3, PostgreSQL, Snowflake, and any other connected source in a single SQL query.

Note: Dremio's Db2 connector supports Db2 for LUW (Linux, UNIX, and Windows). Db2 for z/OS and Db2 for i are not directly supported. If your Db2 instance runs on z/OS or IBM i, you may need to set up a Db2 Connect gateway or replicate to a Db2 LUW instance.

Why Db2 Users Need Dremio

Access Db2 Data Without IBM Middleware

Accessing Db2 analytically typically requires IBM DataStage, IBM Cognos, or custom JDBC applications. These tools are expensive, require specialized skills, and create vendor lock-in. Dremio provides a vendor-neutral SQL layer that connects Db2 to any BI tool (Tableau, Power BI, Looker) via Arrow Flight or ODBC — no IBM middleware needed.

Federate Mainframe Data with Cloud Sources

Your core banking transactions are in Db2, but your digital banking data is in PostgreSQL on AWS, your customer support data is in MongoDB, and your regulatory data is in S3. Without a federation layer, building a 360-degree customer view requires extracting data from each source into a common warehouse. Dremio queries each in place and joins them at query time.

Incremental Modernization

Migrating off Db2 is a multi-year, high-risk project that many organizations cannot undertake. Dremio lets you modernize incrementally: start by querying Db2 through Dremio alongside cloud sources, then gradually migrate specific datasets to Iceberg tables. The migration happens over time, with Db2 continuing to serve critical transactional workloads throughout.

Cost Reduction

IBM mainframe MIPS pricing means every Db2 query consumes expensive compute capacity. Dremio's Reflections cache analytical results so repeated queries don't consume Db2 MIPS. This can meaningfully reduce mainframe compute costs for organizations with heavy analytical workloads against Db2.

AI on Legacy Data

Db2 holds decades of institutional data — customer histories, transaction patterns, risk assessments. Dremio's AI capabilities make this data accessible to non-technical users and external AI tools, unlocking insights trapped in mainframe systems.

Prerequisites

Db2 LUW hostname or IP address — the Db2 server
Port — default 50000 for Db2 LUW
Database name — the Db2 database you want to connect
Username and password — Db2 user with SELECT privileges on the schemas/tables to query
Network access — port 50000 reachable from Dremio Cloud
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect Db2 to Dremio Cloud

1. Add the Source

Click "+" and select IBM Db2 from the database source types.

2. Configure Connection

Name: Descriptive identifier (e.g., db2-banking or mainframe-core).
Host: Db2 server hostname or IP address.
Port: Default 50000.
Database: The Db2 database name.

3. Set Authentication

Master Credentials (username/password) or Secret Resource URL (AWS Secrets Manager).

4. Configure Advanced Options

Setting	Purpose	Default
Record fetch size	Rows per batch from Db2	200
Maximum Idle Connections	Connection pool management	8
Connection Idle Time (s)	Seconds before idle connections close	60
Encrypt Connection	Enable SSL/TLS	Off
Connection Properties	Custom JDBC parameters	None

5. Configure Reflection Refresh and Metadata, Save

Query Db2 Data

-- Query core banking accounts
SELECT
  account_id,
  customer_id,
  account_type,
  current_balance,
  last_transaction_date
FROM "db2-banking".BANK.ACCOUNTS
WHERE account_type = 'SAVINGS' AND current_balance > 10000
ORDER BY current_balance DESC;

-- Transaction analysis
SELECT
  account_type,
  DATE_TRUNC('month', transaction_date) AS month,
  COUNT(*) AS transaction_count,
  SUM(transaction_amount) AS total_amount,
  AVG(transaction_amount) AS avg_amount
FROM "db2-banking".BANK.TRANSACTIONS
WHERE transaction_date >= '2024-01-01'
GROUP BY account_type, DATE_TRUNC('month', transaction_date)
ORDER BY 1, 2;

Federate Db2 with Cloud Sources

-- Join Db2 core banking with PostgreSQL digital banking and S3 support data
SELECT
  a.account_id,
  a.current_balance,
  pg.last_login_date,
  pg.mobile_transactions_30d,
  s3.support_tickets_open,
  CASE
    WHEN a.current_balance > 100000 AND pg.mobile_transactions_30d > 10 THEN 'High Value - Digitally Active'
    WHEN a.current_balance > 100000 THEN 'High Value - Branch Preferred'
    WHEN pg.mobile_transactions_30d > 20 THEN 'Digital Native'
    ELSE 'Standard'
  END AS customer_segment
FROM "db2-banking".BANK.ACCOUNTS a
LEFT JOIN "postgres-digital".public.customer_activity pg ON a.customer_id = pg.customer_id
LEFT JOIN "s3-support".tickets.customer_tickets s3 ON a.customer_id = s3.customer_id
ORDER BY a.current_balance DESC;

Mainframe banking data joins with cloud application data in a single query — no CDC, no data extraction.

Build a Semantic Layer

CREATE VIEW analytics.gold.customer_banking360 AS
SELECT
  a.customer_id,
  a.account_type,
  a.current_balance,
  pg.customer_name,
  pg.email,
  CASE
    WHEN a.current_balance > 250000 THEN 'Private Banking'
    WHEN a.current_balance > 50000 THEN 'Premium'
    WHEN a.current_balance > 10000 THEN 'Standard'
    ELSE 'Basic'
  END AS service_tier,
  DATEDIFF(DAY, a.last_transaction_date, CURRENT_DATE) AS days_since_last_transaction
FROM "db2-banking".BANK.ACCOUNTS a
LEFT JOIN "postgres-digital".public.customers pg ON a.customer_id = pg.customer_id;

In the Catalog, click Edit → Generate Wiki and Generate Tags. Create descriptions like: "customer_banking360: Combines mainframe core banking data with digital channel activity to provide a complete customer view for relationship management."

AI-Powered Analytics on Db2 Data

Dremio AI Agent

The AI Agent transforms access to mainframe data. Instead of needing a Db2 DBA to write queries against complex schemas, a relationship manager asks "Show me all Private Banking customers who haven't transacted in 30 days" and gets accurate results from the semantic layer. The Agent reads your wiki descriptions to understand what "Private Banking" (balance > $250K) and "days_since_last_transaction" mean.

This democratizes access to decades of mainframe data that was previously accessible only through COBOL reports or specialized IBM tools.

Dremio MCP Server

Connect Claude or ChatGPT to your Db2 data:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs for your AI client
Connect via mcp.dremio.cloud/mcp/{project_id}

A compliance officer asks Claude "Show me all accounts with balances over $100K and no transactions in 60 days for our dormancy review" and gets a governed, accurate report from Db2 — without knowing Db2 table structures.

AI SQL Functions

-- Classify account risk with AI
SELECT
  customer_id,
  service_tier,
  current_balance,
  days_since_last_transaction,
  AI_CLASSIFY(
    'Based on these banking patterns, classify the account dormancy risk',
    'Tier: ' || service_tier || ', Balance: $' || CAST(current_balance AS VARCHAR) || ', Days Inactive: ' || CAST(days_since_last_transaction AS VARCHAR),
    ARRAY['Active', 'At Risk', 'Potentially Dormant', 'Dormant']
  ) AS dormancy_risk
FROM analytics.gold.customer_banking360
WHERE days_since_last_transaction > 30;

-- Generate relationship manager talking points
SELECT
  customer_name,
  service_tier,
  AI_GENERATE(
    'Write a one-sentence talking point for a relationship manager reaching out to this customer',
    'Customer: ' || customer_name || ', Tier: ' || service_tier || ', Balance: $' || CAST(current_balance AS VARCHAR) || ', Inactive Days: ' || CAST(days_since_last_transaction AS VARCHAR)
  ) AS outreach_talking_point
FROM analytics.gold.customer_banking360
WHERE service_tier = 'Private Banking' AND days_since_last_transaction > 14;

Reflections for Mainframe Cost Reduction

Every query against Db2 on a mainframe consumes MIPS. Create Reflections to cache frequently accessed analytics:

Navigate to the view in the Catalog
Click the Reflections tab
Choose Raw Reflection or Aggregation Reflection
Select columns and set the Refresh Interval — hourly for active accounts, daily for historical analysis
Click Save

Dashboard and reporting queries hit Reflections instead of Db2, significantly reducing mainframe compute consumption. A compliance dashboard that refreshes every 15 minutes generates zero Db2 MIPS after the Reflection is built.

Governance on Db2 Data

Banking, insurance, and government organizations have strict data governance requirements. Dremio's Fine-Grained Access Control (FGAC) extends Db2 security to every connected source:

Column masking: Mask account balances, SSNs, and transaction amounts from specific roles. A marketing analyst sees customer segments but not financial data.
Row-level filtering: Branch-level access control — a branch manager sees only their branch's accounts.
Unified policies: Same governance applies across Db2, PostgreSQL, S3, and all other connected sources.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server — meeting regulatory requirements for data access control.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Tableau: Dremio connector for direct Arrow Flight access to mainframe data
Power BI: Dremio ODBC driver or native connector — no IBM middleware
Python/Pandas: pyarrow.flight for programmatic access to Db2 data
Looker: Connect via JDBC
dbt: dbt-dremio for SQL-based transformations on Db2 data

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Db2 data from their IDE. Ask Copilot "Show me dormant high-value accounts from Db2" and get SQL generated using your semantic layer — without knowing Db2 table schemas or COBOL naming conventions.

When to Keep Data in Db2 vs. Migrate

Keep in Db2: Active transactional data for applications, data with COBOL program dependencies, regulatory data that must maintain system of record status, data subject to mainframe-specific compliance requirements.

Migrate to Iceberg: Historical transaction archives (closed accounts, prior fiscal years), data consumed by non-mainframe tools, datasets where mainframe MIPS cost exceeds analytical value, archived data for long-term retention.

For data staying in Db2, create manual Reflections to reduce MIPS consumption. For migrated Iceberg data, Dremio provides automatic compaction, time travel, Autonomous Reflections, and dramatically lower storage costs.

Db2 Character Encoding and Data Types

Db2 uses EBCDIC encoding on mainframes and ASCII/UTF-8 on LUW platforms. When connecting through Dremio:

EBCDIC to UTF-8: Db2 for LUW handles character conversion automatically — Dremio receives standard Unicode data
GRAPHIC/VARGRAPHIC: Double-byte character columns map to VARCHAR in Dremio
DECIMAL/NUMERIC: Db2's fixed-point types map to Dremio's DECIMAL with matching precision/scale
DATE/TIME/TIMESTAMP: Standard mapping — Db2 timestamps map to Dremio TIMESTAMP

Regulatory Compliance Patterns

Banking, insurance, and government organizations have strict data retention and access requirements. Dremio addresses these:

Requirement	Dremio Feature
Data residency	Query data in place — no cross-border data movement
Access auditing	Query logs track who queried what data
Column-level security	FGAC column masking hides sensitive fields
Row-level security	FGAC row filtering restricts data by user role
Data retention	Time travel on Iceberg tables provides point-in-time access

Mainframe Modernization Roadmap

Use Dremio as the bridge in a multi-year mainframe modernization:

Phase 1 (Months 1-3): Connect Db2 to Dremio Cloud. Create Reflections to offload analytical queries.
Phase 2 (Months 4-6): Build a semantic layer over Db2 data. Enable AI Agent and MCP Server for business users.
Phase 3 (Months 7-12): Identify high-value datasets for migration to Iceberg. Use CREATE TABLE AS SELECT to migrate.
Phase 4 (Year 2+): Gradually migrate remaining datasets as mainframe contracts renew. Db2 focus narrows to core OLTP.

Throughout the process, users experience no disruption — they continue using the same semantic layer views. Only the underlying data sources change.

Get Started

Db2 users can modernize analytics without migrating off the mainframe — federate, govern, accelerate, and AI-enable decades of institutional data through Dremio Cloud. Start with Reflections to offload analytical queries from Db2, then progressively build a semantic layer that makes legacy data accessible to modern AI tools and business users.

Try Dremio Cloud free for 30 days and connect your Db2 databases.

Connect Microsoft SQL Server to Dremio Cloud: Federate Enterprise Data Without ETL

Alex Merced — Sun, 01 Mar 2026 12:00:00 GMT

Microsoft SQL Server is one of the most widely deployed enterprise databases in the world. ERP systems, CRM platforms, financial applications, and custom business applications run on SQL Server across on-premises data centers and Azure cloud deployments. But connecting SQL Server data to a modern analytics platform typically requires building ETL pipelines, managing SSIS packages, or purchasing additional SQL Server Enterprise licenses for analytics workloads.

Dremio Cloud connects directly to SQL Server and queries it alongside S3, PostgreSQL, Snowflake, BigQuery, MongoDB, and every other connected source in a single SQL query. You don't need to extract data from SQL Server, build staging tables, or manage nightly ETL jobs. Dremio reads SQL Server in place, applies governance, and accelerates repeated queries with Reflections.

SQL Server licensing is notoriously expensive — Enterprise edition costs tens of thousands of dollars per core. Running analytical queries directly against production SQL Server instances consumes CPU capacity that's licensed for transactional workloads. Dremio's Reflections cache analytical results, offloading read-heavy queries from SQL Server and potentially allowing organizations to reduce their SQL Server core count or downgrade from Enterprise to Standard edition.

Why SQL Server Users Need Dremio

Escape Linked Server Limitations

SQL Server's linked servers provide basic federation, but they're limited: poor cross-platform support (try linking to MongoDB or BigQuery), no query optimization across links, no governance layer, and performance degrades with large result sets. Dremio's federation engine is purpose-built for cross-source queries — it pushes predicates to each source, optimizes join strategies, and handles large-scale data movement efficiently.

Reduce SQL Server License Costs

SQL Server Enterprise licensing is expensive — especially when analytical workloads compete with transactional OLTP operations for CPU and memory. Dremio's Reflections offload repeated analytical queries from SQL Server: dashboard refreshes, scheduled reports, and ad-hoc exploration hit cached Reflections instead of SQL Server. This can reduce the SQL Server resources dedicated to analytics, potentially allowing you to downgrade from Enterprise to Standard edition or reduce core counts.

Multi-Cloud, Multi-Database Analytics

Your SQL Server holds ERP data, but your data lake is on S3, your marketing data is in Google BigQuery, and your modern applications use PostgreSQL. Without Dremio, combining these requires SSIS packages, Azure Data Factory, or custom ETL for each source. Dremio queries all of them in a single SQL statement.

Unified Governance Beyond Windows

SQL Server has Windows Authentication and SQL Logins, but these don't apply to your S3 data lake, BigQuery, or PostgreSQL. Dremio's Fine-Grained Access Control applies column masking and row-level filtering consistently across SQL Server and every other connected source.

AI Analytics on Enterprise Data

SQL Server stores decades of business data — financial records, customer histories, inventory movements. Dremio's AI Agent, MCP Server, and AI SQL Functions make that historical data queryable by natural language and enrichable by AI, unlocking insights that would otherwise require a data analyst with deep institutional knowledge.

Prerequisites

SQL Server hostname or IP address
Port — default 1433
Database name
Username and password (SQL Authentication) — user needs SELECT permissions on target schemas and tables
Network access — port 1433 must be reachable from Dremio Cloud. For on-premises SQL Server, configure VPN or firewall rules
Dremio Cloud account — sign up free for 30 days with $400 in compute credits

Step-by-Step: Connect SQL Server to Dremio Cloud

1. Add the Source

Click "+" in the Dremio console and select Microsoft SQL Server.

2. Configure Connection

Name: Descriptive identifier (e.g., sqlserver-erp or production-db).
Host: SQL Server hostname or IP address.
Port: Default 1433.
Database: The database name to connect to.

3. Set Authentication

Enter SQL Authentication credentials (username/password) or use Secret Resource URL for centralized credential management via AWS Secrets Manager.

4. Configure Advanced Options

Setting	Purpose	Default
Record fetch size	Rows per batch from SQL Server	200
Maximum Idle Connections	Connection pool management	8
Connection Idle Time (s)	Seconds before idle connections close	60
Encrypt Connection	Enable SSL/TLS	Off
SSL Verification	Verify SSL server certificate	Off
Hostname in Certificate	Expected hostname in SSL certificate	None
Connection Properties	Custom JDBC parameters	None

5. Configure Reflection Refresh and Metadata, Save

Query SQL Server Data

-- Query ERP inventory data
SELECT
  product_id,
  product_name,
  warehouse_location,
  quantity_on_hand,
  reorder_point
FROM "sqlserver-erp".dbo.products
WHERE quantity_on_hand < reorder_point
ORDER BY quantity_on_hand ASC;

-- Financial reporting
SELECT
  department_code,
  account_category,
  fiscal_quarter,
  SUM(actual_amount) AS actual_spend,
  SUM(budget_amount) AS budgeted,
  ROUND((SUM(actual_amount) - SUM(budget_amount)) / NULLIF(SUM(budget_amount), 0) * 100, 1) AS variance_pct
FROM "sqlserver-erp".finance.budget_actuals
WHERE fiscal_year = 2024
GROUP BY department_code, account_category, fiscal_quarter
ORDER BY ABS(SUM(actual_amount) - SUM(budget_amount)) DESC;

Federate SQL Server with Other Sources

-- Join SQL Server ERP with PostgreSQL CRM and S3 marketing data
SELECT
  ss.product_name,
  ss.quantity_on_hand,
  pg.total_orders,
  pg.avg_order_value,
  s3.click_through_rate,
  CASE
    WHEN pg.total_orders > 100 AND ss.quantity_on_hand < 50 THEN 'Reorder - High Demand'
    WHEN pg.total_orders < 10 AND ss.quantity_on_hand > 500 THEN 'Overstock - Reduce'
    ELSE 'Normal'
  END AS inventory_action
FROM "sqlserver-erp".dbo.products ss
LEFT JOIN (
  SELECT product_id, COUNT(*) AS total_orders, AVG(order_value) AS avg_order_value
  FROM "postgres-crm".public.orders
  WHERE order_date >= '2024-01-01'
  GROUP BY product_id
) pg ON ss.product_id = pg.product_id
LEFT JOIN "s3-marketing".analytics.product_clicks s3 ON ss.product_id = s3.product_id
WHERE ss.quantity_on_hand < ss.reorder_point OR pg.total_orders > 100
ORDER BY pg.total_orders DESC;

Build a Semantic Layer

CREATE VIEW analytics.gold.inventory_management AS
SELECT
  p.product_id,
  p.product_name,
  p.warehouse_location,
  p.quantity_on_hand,
  p.reorder_point,
  CASE
    WHEN p.quantity_on_hand = 0 THEN 'Out of Stock'
    WHEN p.quantity_on_hand < p.reorder_point * 0.5 THEN 'Critical'
    WHEN p.quantity_on_hand < p.reorder_point THEN 'Low'
    ELSE 'Adequate'
  END AS stock_status,
  ROUND(p.quantity_on_hand * p.unit_cost, 2) AS inventory_value
FROM "sqlserver-erp".dbo.products p;

In the Catalog, click Edit → Generate Wiki and Generate Tags. Create descriptions like: "inventory_management: One row per product showing current stock levels, stock status classification, and estimated inventory value. Use this view to monitor reorder needs."

AI-Powered Analytics on SQL Server Data

Dremio AI Agent

The AI Agent lets operations managers ask "Which products are critically low at the Chicago warehouse?" without writing SQL. The Agent reads your wiki descriptions, understands "Critical" means stock below 50% of reorder point, and generates accurate queries. This is transformative for SQL Server environments where tribal knowledge about table schemas and column meanings lives in senior employees' heads.

Dremio MCP Server

Connect Claude or ChatGPT to your SQL Server data:

Create a Native OAuth app in Dremio Cloud
Configure redirect URLs for your AI client
Connect via mcp.dremio.cloud/mcp/{project_id}

A warehouse manager asks Claude "Show me all products that need reordering, sorted by how critical the shortage is" and gets actionable results from the semantic layer over SQL Server — no SQL, no SSMS.

AI SQL Functions

-- Generate reorder recommendations with AI
SELECT
  product_name,
  stock_status,
  quantity_on_hand,
  reorder_point,
  AI_GENERATE(
    'Write a one-sentence reorder recommendation based on inventory status',
    'Product: ' || product_name || ', Stock: ' || CAST(quantity_on_hand AS VARCHAR) || ', Reorder Point: ' || CAST(reorder_point AS VARCHAR) || ', Status: ' || stock_status
  ) AS reorder_recommendation
FROM analytics.gold.inventory_management
WHERE stock_status IN ('Critical', 'Out of Stock');

-- Classify financial variances
SELECT
  department_code,
  variance_pct,
  AI_CLASSIFY(
    'Based on the budget variance, classify the financial risk level',
    'Department: ' || department_code || ', Variance: ' || CAST(variance_pct AS VARCHAR) || '%',
    ARRAY['On Track', 'Minor Variance', 'Significant Overspend', 'Critical Overspend']
  ) AS financial_risk
FROM (
  SELECT department_code, ROUND((SUM(actual_amount) - SUM(budget_amount)) / NULLIF(SUM(budget_amount), 0) * 100, 1) AS variance_pct
  FROM "sqlserver-erp".finance.budget_actuals
  WHERE fiscal_year = 2024
  GROUP BY department_code
);

Reflections for Performance

SQL Server Enterprise charges per-core licensing. Offloading analytical queries to Reflections reduces compute pressure on SQL Server cores:

Navigate to the view in the Catalog
Click the Reflections tab
Choose Raw Reflection or Aggregation Reflection
Select columns and set the Refresh Interval — for ERP data updated throughout the day, hourly; for financial data, match to reporting cycles
Click Save

BI tools get sub-second response times from Reflections. SQL Server focuses on transactional OLTP workloads. A financial dashboard refreshing every 15 minutes generates zero SQL Server load after the Reflection is built.

Governance Across SQL Server and Other Sources

Dremio's Fine-Grained Access Control (FGAC) extends SQL Server security to every connected source:

Column masking: Mask financial data, salary details, or PII from specific roles. A warehouse manager sees inventory levels but not cost data.
Row-level filtering: Regional managers see only their region's data. Department heads see only their department.
Unified policies: Same governance applies across SQL Server, PostgreSQL, S3, and all other sources.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC for SQL Server data:

Power BI: Dremio native connector — ideal for Microsoft-centric organizations
Tableau: Dremio connector for direct Arrow Flight access
Python/Pandas: pyarrow.flight for programmatic data access
Looker: Connect via JDBC
dbt: dbt-dremio adapter for transformation workflows

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query SQL Server data from their IDE. Ask Copilot "Show me products below reorder point at the Chicago warehouse" and get SQL generated from your semantic layer.

When to Keep Data in SQL Server vs. Migrate

Keep in SQL Server: Transactional data for active applications, data with stored procedures and triggers, operational systems that depend on SQL Server features (SSRS, SSIS, linked servers).

Migrate to Iceberg: Historical records and archives, reporting data, data consumed by non-SQL-Server tools, datasets where SQL Server per-core licensing cost exceeds analytical value. Migrated Iceberg tables get Dremio's automatic compaction, time travel, and Autonomous Reflections.

For data staying in SQL Server, create manual Reflections. For migrated Iceberg data, Dremio handles optimization automatically.

Query Pushdown to SQL Server

Dremio's federation engine optimizes cross-source queries by pushing operations to SQL Server whenever possible:

Filter pushdown: WHERE clauses are pushed to SQL Server, so only matching rows are transferred
Projection pushdown: Only the columns referenced in your query are requested from SQL Server
Aggregate pushdown: SUM, COUNT, AVG operations can be executed on SQL Server when the full query allows

This minimizes data transfer between SQL Server and Dremio, reducing network traffic and improving query performance.

ERP Integration Patterns

SQL Server frequently powers ERP systems (Microsoft Dynamics, custom internal ERPs). Dremio enables analytics that combine ERP data with external sources:

SQL Server (ERP)	External Source	Analytics Use Case
Inventory levels	S3 demand forecasts	Automated reorder predictions
Purchase orders	PostgreSQL supplier data	Supplier performance scoring
Financial actuals	BigQuery market data	Revenue benchmarking
Customer accounts	MongoDB support tickets	Churn risk assessment

These cross-source analytics are impossible with SQL Server alone and traditionally require SQL Server Integration Services (SSIS) to build ETL pipelines. Dremio eliminates this requirement entirely.

SQL Server Always Encrypted and SSL

Dremio supports SSL/TLS connections to SQL Server. For databases using Always Encrypted columns, be aware that Dremio reads the encrypted values — decryption requires the Column Master Key, which is managed by the application. For analytical workloads, consider creating views on the SQL Server side that expose non-encrypted analytical summaries.

Get Started

SQL Server users can federate enterprise data, reduce license costs, deploy AI analytics, and apply unified governance across their entire data estate. Start by connecting your primary SQL Server instance to Dremio Cloud. Create Reflections on your most-queried reporting tables to offload analytical queries from SQL Server immediately, reducing CPU load and freeing licensed cores for transactional workloads.

Try Dremio Cloud free for 30 days and connect your SQL Server instances.

Extract Structured Data from Text with Dremio's AI_GENERATE Function

Alex Merced — Sun, 01 Mar 2026 11:00:00 GMT

Unstructured text is the most underused data in most organizations. Customer emails sit in inboxes. Contract notes live in text fields. Meeting summaries exist as free-text columns in CRM systems. The information is there, but it's locked inside prose that SQL can't filter, join, or aggregate.

Dremio's AI_GENERATE function breaks that lock. It sends unstructured text to an LLM and returns structured rows with typed columns. You define the output schema directly in SQL, and the LLM extracts the fields you specify. An email becomes a row with sender, subject, priority, and action_items columns. A contract note becomes a row with party_name, contract_value, start_date, and terms.

This tutorial builds a complete document processing pipeline in a fresh Dremio Cloud account. You'll create sample email and contract data, build a medallion architecture, and use AI_GENERATE to extract structured fields from free text. A separate section covers using AI_GENERATE with LIST_FILES to process unstructured files (PDFs, text files) stored in object storage.

What You'll Build

By the end of this tutorial, you'll have:

A dataset with 50+ raw emails and 25+ contract notes containing free-text descriptions
Bronze views that standardize raw data
Silver views that join emails with contract information
Gold views that use AI_GENERATE with WITH SCHEMA to extract structured fields from text
Materialized Iceberg tables that persist extracted data for downstream analytics
An understanding of how to combine AI_GENERATE with LIST_FILES for file-based extraction

Prerequisites

Dremio Cloud account — sign up free for 30 days with $400 in compute credits
AI enabled — go to Admin → Project Settings → Preferences → AI section and enable AI features
Model Provider configured — Dremio provides a hosted LLM by default, or connect your own (OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI)

Note: Tables in the built-in Open Catalog use folder.subfolder.table_name without a catalog prefix. External sources use source_name.schema.table_name.

Understanding AI_GENERATE

AI_GENERATE is the most powerful of Dremio's AI SQL functions because it returns structured data from unstructured input. The function signature:

AI_GENERATE(
  [model_name VARCHAR,]
  prompt VARCHAR,
  target_data VARCHAR
  [WITH SCHEMA (field_name DATA_TYPE, ...)]
) → ROW | VARCHAR

Parameters:

model_name (optional) — specify a model like 'openai.gpt-4o'. Format is modelProvider.modelName.
prompt — the extraction instruction telling the LLM what fields to find in the target data.
target_data — the unstructured text column to process. This is usually a column from your table containing emails, notes, descriptions, or document content.
WITH SCHEMA (optional but recommended) — defines the output structure as a ROW type with named, typed columns. Without it, AI_GENERATE returns a VARCHAR (plain text). With it, you get a ROW that you can expand using dot notation.

The WITH SCHEMA clause is what makes AI_GENERATE different from AI_COMPLETE. Instead of getting free-form text back, you get a typed row where each field is a column you defined, ready for filtering, joining, and aggregating.

ROW Type Output

When you use WITH SCHEMA, the result is a ROW type. Access individual fields with dot notation:

SELECT
  result.sender,
  result.priority,
  result.action_items
FROM (
  SELECT AI_GENERATE(
    'Extract key information from this email',
    email_body
    WITH SCHEMA (sender VARCHAR, priority VARCHAR, action_items VARCHAR)
  ) AS result
  FROM emails
);

Step 1: Create Your Folder Structure

Open the SQL Runner:

CREATE FOLDER IF NOT EXISTS aigenerateexp;
CREATE FOLDER IF NOT EXISTS aigenerateexp.document_data;
CREATE FOLDER IF NOT EXISTS aigenerateexp.bronze;
CREATE FOLDER IF NOT EXISTS aigenerateexp.silver;
CREATE FOLDER IF NOT EXISTS aigenerateexp.gold;

Step 2: Seed Your Sample Data

Raw Emails Table

This table simulates emails stored in a CRM system. Each email has a free-text body that contains multiple pieces of information: who sent it, what they're asking about, how urgent it is, and what action is needed. Extracting these fields manually would require a human to read each email. AI_GENERATE automates this.

CREATE TABLE aigenerateexp.document_data.raw_emails (
  email_id INT,
  received_date DATE,
  email_body VARCHAR
);

INSERT INTO aigenerateexp.document_data.raw_emails VALUES
(1, '2025-09-01', 'Hi team, this is Sarah Chen from Acme Corp. We need to urgently discuss the renewal of our enterprise license. Our current contract expires on October 15th and we want to add 200 additional seats. Can someone from your licensing team contact me by end of week? My direct line is 555-0142. Thanks, Sarah'),
(2, '2025-09-02', 'To whom it may concern, I am writing to report a critical production outage affecting our CloudSync deployment. All file synchronization stopped at 3:47 AM EST this morning. Over 500 users are impacted. We need immediate escalation to your Level 3 support team. This is a P1 issue per our SLA terms. Regards, James Rodriguez, VP of IT, Global Industries'),
(3, '2025-09-03', 'Hello, my name is Emily Watson and I am the procurement manager at TechStart Inc. We are evaluating DataVault Enterprise for our compliance requirements. Could you send me pricing information for a 3-year commitment with 150 users? Also interested in the SOC 2 audit documentation. Our budget review is scheduled for next month so no rush. Best, Emily'),
(4, '2025-09-05', 'URGENT: Our QuickReport installation has been down for 6 hours. Dashboard presentations to the board of directors are in 2 hours. We need the reporting engine restored immediately. Client: MegaCorp Financial. Contact: David Kim, CFO. Phone: 555-0198. This is affecting our quarterly earnings presentation.'),
(5, '2025-09-06', 'Hi there, I wanted to share some positive feedback. Your DevPipeline product has reduced our deployment time from 45 minutes to under 3 minutes. Our engineering team of 80 developers is very happy with the migration. We are considering expanding to our European offices next quarter. Great product! - Michael Brown, CTO, CloudNine Software'),
(6, '2025-09-08', 'Dear Support, we recently purchased MailForge for our marketing team but are having trouble with the SMTP relay configuration. Emails are being flagged as spam by Gmail and Outlook recipients. Our deliverability rate dropped from 98% to 62% after switching to MailForge. This is not urgent but needs resolution by end of month before our holiday campaign launches. Sincerely, Lisa Park, Marketing Director, RetailPlus'),
(7, '2025-09-10', 'To the sales team: We are a healthcare organization looking for a HIPAA-compliant backup solution. We evaluated CloudBackup but have concerns about the BAA terms in section 4.2. Can your legal team review our proposed amendments? We handle PHI for approximately 50000 patients. Timeline: need decision by November 1st. Contact: Dr. Anna Kowalski, Chief Medical Information Officer, Metro Health System'),
(8, '2025-09-11', 'I am writing to formally request cancellation of our HelpDesk360 subscription effective immediately. The product has not met our expectations. Response routing is inaccurate, the knowledge base search returns irrelevant results, and we have experienced 3 unplanned outages in the past month. Please process our refund for the remaining 8 months on our annual contract. Robert Taylor, Operations Director, ServiceFirst Ltd'),
(9, '2025-09-12', 'Quick question: does FormBuilder support WCAG 2.1 AA compliance for government forms? We are a state agency and this is a hard requirement for procurement. If yes, can you point me to the VPAT documentation? Thanks, Maria Garcia, Accessibility Coordinator, State of California Department of Technology'),
(10, '2025-09-14', 'Hi, our team has been using TeamBoard for 6 months and we love it. However we really need a way to export Gantt charts to PDF while preserving the formatting. The current export flattens all the dependency lines and makes the chart unreadable. Is this on your roadmap? Our PMO presents these charts to clients weekly. Tom Williams, PMO Lead, ConsultCo'),
(11, '2025-09-15', 'INCIDENT REPORT: At approximately 14:22 UTC our SecureSign production environment began experiencing signature verification failures. Approximately 340 pending documents across 12 customer accounts are affected. Root cause appears to be an expired intermediate SSL certificate in your signing chain. We need immediate remediation. Kevin Thompson, Security Engineer, LegalTech Partners'),
(12, '2025-09-17', 'Dear team, we operate DataStream to process 2TB of Kafka events daily. Starting last week we noticed exactly-once processing guarantees are failing intermittently. Approximately 0.3% of events are being duplicated in our downstream Postgres sink. This is causing financial reconciliation errors in our billing system. Medium priority but needs attention within 2 weeks. Jennifer Lee, Senior Data Engineer, FinServ Analytics'),
(13, '2025-09-18', 'I would like to schedule a product demo of AdOptimizer for our digital marketing agency. We manage ad spend for 45 clients across Google Ads Facebook and LinkedIn totaling approximately 2.5M monthly. Currently using a competitor but unhappy with the attribution modeling accuracy. When is your team available next week? Chris Martinez, Founder, DigitalEdge Agency'),
(14, '2025-09-20', 'Hi, we just completed our evaluation of ContractManager and would like to proceed with a purchase for 75 seats. We need the Salesforce integration enabled from day one. Our legal team processes roughly 200 contracts per month and we are currently tracking everything in spreadsheets. What is the implementation timeline? Rachel Adams, General Counsel, NovaTech Industries'),
(15, '2025-09-22', 'Attention: We detected unauthorized API access attempts against our LogInsight deployment between 2AM and 4AM EST today. The requests originated from IP addresses in a known threat intelligence database. While our firewall blocked the attempts, we want to understand if LogInsight has additional rate limiting or IP blocking capabilities we should enable. Mark Allen, CISO, DataShield Corp'),
(16, '2025-09-24', 'To billing department: Our organization PayFlow account 8847291 shows a currency conversion fee of 2.8% on GBP transactions. Our contract specifies a 1.5% rate for all EUR and GBP conversions. Please correct this billing discrepancy retroactively for September transactions totaling approximately 45000 GBP. Amanda Clark, Treasury Manager, EuroCommerce BV'),
(17, '2025-09-25', 'Hello, we have been running ChatAssist for our e-commerce customer support and the intent classification accuracy is excellent at around 94%. However we need to add support for Portuguese and Thai languages. Our customer base expanded to Brazil and Thailand this quarter. Is the multi-language add-on available for our current plan tier? Steven Moore, VP Customer Experience, GlobalShop'),
(18, '2025-09-27', 'I am the HR director at a 2000-employee manufacturing company. We need SchedulePro to handle complex shift patterns including rotating shifts split shifts and on-call schedules. Our current system cannot handle the overtime calculations required by state-specific labor laws in California New York and Texas. Can SchedulePro handle multi-state labor law compliance? Catherine Hall, HR Director, PrecisionMfg Inc'),
(19, '2025-09-28', 'Feature request: DesignHub needs better support for design tokens and component variables. When we update a color in our design system it should propagate to all linked components across all projects automatically. Currently we have to manually update 200+ components which defeats the purpose of a design system. Otherwise great product. Brian Harris, Design Systems Lead, PixelPerfect Studio'),
(20, '2025-09-30', 'Dear sales, I am reaching out on behalf of a consortium of 12 regional banks looking for a unified API management solution. We collectively process 4.2M API requests daily and need a solution that supports PSD2 compliance including strong customer authentication and secure communication. Can we arrange a meeting with your banking vertical team? Daniel Wilson, Technology Director, Regional Banking Alliance'),
(21, '2025-10-01', 'Hi, quick update on our InventoryTrack implementation. The barcode scanning module is working perfectly in our main warehouse but the multi-warehouse sync is showing a 15-minute delay between facilities. For perishable goods this delay causes stock discrepancies. Can we reduce the sync interval to real-time? Sophia Nguyen, Warehouse Operations Manager, FreshFoods Distribution'),
(22, '2025-10-03', 'To the product team at CloudSync: I have been a loyal customer for 3 years and want to share feedback. The recent UI redesign is excellent but the new settings menu is confusing. I cannot find the bandwidth throttling option which I use daily. Please make frequently used settings more accessible. Otherwise love the product and have recommended it to 5 colleagues. Laura Jackson, IT Consultant'),
(23, '2025-10-05', 'CRITICAL: Our DataVault encryption at rest failed an internal penetration test. The AES-256 implementation is using ECB mode instead of CBC or GCM for blocks larger than 16 bytes. This is a known vulnerability pattern. We need confirmation that this will be patched before our next compliance audit on November 15th. Michelle Lopez, Information Security Analyst, SecureBank NA'),
(24, '2025-10-06', 'Hello, I am a professor at MIT and we use QuickReport for our research data visualization. We are interested in an academic licensing program. Our department has 35 researchers and 120 graduate students who would benefit from the tool. Is there an education discount available? Dr. Jessica Young, Department of Data Science, MIT'),
(25, '2025-10-08', 'Support ticket follow-up: Our MailForge DKIM configuration issue ticket 4421 was marked resolved but we are still failing DMARC checks from Yahoo and AOL. The DKIM record appears correctly in DNS but the selector value does not match what MailForge sends in the email headers. Need this escalated back to engineering. Andrew White, Email Administrator, NewsMedia Group');

Contract Notes Table

This table simulates free-text contract summaries written by account managers. Each note contains key contract details buried in natural language that AI_GENERATE will extract into structured columns.

CREATE TABLE aigenerateexp.document_data.contract_notes (
  note_id INT,
  account_manager VARCHAR,
  note_date DATE,
  note_text VARCHAR
);

INSERT INTO aigenerateexp.document_data.contract_notes VALUES
(1, 'Patricia Moore', '2025-09-01', 'Closed deal with Acme Corp for CloudSync Pro enterprise license. 500 seats at $22/seat/month for 3-year term. Total contract value $396,000. Includes premium support and 99.99% SLA. Renewal auto-triggers 90 days before expiration. Key contact: Sarah Chen, VP Engineering.'),
(2, 'Marcus Johnson', '2025-09-04', 'MegaCorp Financial signed for QuickReport Premium. 200 users, 2-year commitment at $42/user/month. TCV $201,600. Custom integration with their Bloomberg terminal data feed required. Implementation starts Oct 1st. Executive sponsor: David Kim, CFO.'),
(3, 'Patricia Moore', '2025-09-08', 'Renewal discussion with Global Industries for DataVault Enterprise. Current contract of 350 seats expires Dec 31. They want to expand to 600 seats and add the healthcare compliance module. Proposed pricing: $75/seat/month for 600 seats, 3-year term. TCV $1,620,000. Pending legal review of updated BAA.'),
(4, 'Sandra Lee', '2025-09-12', 'New customer TechStart Inc closed for DataVault Enterprise. 150 seats, 3-year term at $82/seat/month. TCV $443,880. SOC 2 documentation provided. Implementation timeline: 6 weeks starting Oct 15. Procurement contact: Emily Watson.'),
(5, 'Marcus Johnson', '2025-09-15', 'DigitalEdge Agency signed AdOptimizer Enterprise with custom attribution modeling. 10 managed accounts, $1,499/month flat rate, 1-year term with option to renew. TCV $17,988. Agency plans to expand to 45 accounts in Q2 2026. Founder Chris Martinez is very enthusiastic about the attribution improvements.'),
(6, 'Sandra Lee', '2025-09-18', 'NovaTech Industries purchased ContractManager Professional. 75 seats at $62/seat/month, 2-year term. TCV $111,600. Salesforce integration required for day-one launch. Processing 200+ contracts monthly currently using spreadsheets. General Counsel Rachel Adams leading internal rollout.'),
(7, 'Patricia Moore', '2025-09-20', 'Lost deal: ServiceFirst Ltd cancelling HelpDesk360 subscription. 8 months remaining on annual contract at $54/seat for 100 seats. Refund request of $43,200 pending finance approval. Customer cited routing accuracy issues, knowledge base relevance problems, and 3 outages. Risk of negative public review.'),
(8, '  Marcus Johnson', '2025-09-22', 'Expansion deal with CloudNine Software for DevPipeline. Adding 80 European developer seats to existing 80 US seats. European deployment at $72/seat/month, 2-year aligned with US contract end. Additional TCV $138,240. CTO Michael Brown driving the expansion after successful US rollout.'),
(9, 'Sandra Lee', '2025-09-25', 'State of California DPT evaluating FormBuilder for government forms. WCAG 2.1 AA compliance confirmed. Potential 500-seat deployment at government rate of $15/seat/month, 5-year term. TCV $450,000. Requires VPAT documentation submission to procurement. Long sales cycle expected, 6-9 months.'),
(10, 'Patricia Moore', '2025-09-28', 'EuroCommerce BV billing dispute on PayFlow. Customer contract guarantees 1.5% FX rate on EUR/GBP but system charged 2.8% for September. Estimated overcharge: approximately $900 on 45K GBP volume. Finance investigating root cause. Treasury Manager Amanda Clark expects retroactive correction.'),
(11, 'Marcus Johnson', '2025-10-01', 'Regional Banking Alliance consortium deal for APIGateway Pro. 12 banks, centralized deployment, 4.2M daily API calls. PSD2 compliance required. Proposed tiered pricing based on volume: $15,000/month for the consortium. 3-year term. TCV $540,000. Technology Director Daniel Wilson coordinating across all 12 institutions.'),
(12, 'Sandra Lee', '2025-10-03', 'FreshFoods Distribution requesting InventoryTrack real-time sync upgrade. Current standard sync has 15-min delay between 4 warehouses causing perishable goods discrepancies. Upgrade to real-time tier: additional $20/warehouse/month. Annual incremental revenue: $960. Operations Manager Sophia Nguyen is the champion.'),
(13, 'Patricia Moore', '2025-10-05', 'GlobalShop expansion for ChatAssist multi-language support. Adding Portuguese and Thai to existing English and Spanish deployment. Current contract: 300 seats at $68/seat/month. Multi-language add-on: additional $12/seat/month. Added TCV for remaining 18 months: $64,800. VP Customer Experience Steven Moore confirmed budget approval.'),
(14, 'Marcus Johnson', '2025-10-06', 'PrecisionMfg Inc evaluating SchedulePro for 2000 employees across 3 US states. Complex requirements: rotating shifts, split shifts, on-call, multi-state overtime compliance for CA, NY, TX. Enterprise tier at $12/employee/month, 2-year term. TCV $576,000. HR Director Catherine Hall leading evaluation. POC planned for November.'),
(15, 'Sandra Lee', '2025-10-08', 'MIT academic licensing request for QuickReport. 35 researchers plus 120 graduate students. Academic program pricing: 70% discount, $14.99/seat/month for 155 seats. 1-year renewable. TCV $27,881. Dr. Jessica Young in Department of Data Science. Low revenue but high brand visibility in academic publications.');

Step 3: Build Bronze Views

Bronze views cast dates to timestamps and standardize column names.

CREATE OR REPLACE VIEW aigenerateexp.bronze.v_emails AS
SELECT
  email_id,
  CAST(received_date AS TIMESTAMP) AS received_timestamp,
  email_body
FROM aigenerateexp.document_data.raw_emails;

CREATE OR REPLACE VIEW aigenerateexp.bronze.v_contracts AS
SELECT
  note_id,
  account_manager,
  CAST(note_date AS TIMESTAMP) AS note_timestamp,
  note_text
FROM aigenerateexp.document_data.contract_notes;

Step 4: Build Silver Views

This Silver view provides the unified email data that Gold views will process. At this stage, we simply promote the Bronze view for downstream extraction.

CREATE OR REPLACE VIEW aigenerateexp.silver.v_email_pipeline AS
SELECT
  email_id,
  received_timestamp,
  email_body
FROM aigenerateexp.bronze.v_emails;

Step 5: Build Gold Views with AI_GENERATE

Gold View 1: Email Information Extraction

This is the core use case for AI_GENERATE. Each email contains a sender, their company, the topic, the urgency level, and an action item, but all of this is embedded in free-text prose. The WITH SCHEMA clause tells the LLM exactly what fields to extract and what types to return.

CREATE OR REPLACE VIEW aigenerateexp.gold.v_email_extracted AS
SELECT
  email_id,
  received_timestamp,
  email_body,
  extracted.sender_name,
  extracted.company,
  extracted.topic,
  extracted.urgency,
  extracted.action_required
FROM (
  SELECT
    email_id,
    received_timestamp,
    email_body,
    AI_GENERATE(
      'Extract the following information from this email. If a field is not present, return N/A.',
      email_body
      WITH SCHEMA (
        sender_name VARCHAR,
        company VARCHAR,
        topic VARCHAR,
        urgency VARCHAR,
        action_required VARCHAR
      )
    ) AS extracted
  FROM aigenerateexp.silver.v_email_pipeline
);

The subquery calls AI_GENERATE and aliases the result as extracted. The outer query then expands the ROW using dot notation (extracted.sender_name, extracted.company, etc.). Each field becomes a regular column you can filter, group, or join on.

Gold View 2: Contract Detail Extraction

Contract notes contain structured deal information in natural language. AI_GENERATE extracts the client name, product, seat count, contract value, term, and key contact into individual columns.

CREATE OR REPLACE VIEW aigenerateexp.gold.v_contract_details AS
SELECT
  note_id,
  account_manager,
  note_timestamp,
  note_text,
  details.client_name,
  details.product,
  details.seat_count,
  details.monthly_rate,
  details.contract_term_years,
  details.total_contract_value,
  details.key_contact,
  details.deal_status
FROM (
  SELECT
    note_id,
    account_manager,
    note_timestamp,
    note_text,
    AI_GENERATE(
      'Extract deal information from this contract note. For total_contract_value use only the numeric amount. For deal_status classify as Won, Lost, Pending, or Expansion.',
      note_text
      WITH SCHEMA (
        client_name VARCHAR,
        product VARCHAR,
        seat_count INT,
        monthly_rate DECIMAL(10,2),
        contract_term_years INT,
        total_contract_value DECIMAL(12,2),
        key_contact VARCHAR,
        deal_status VARCHAR
      )
    ) AS details
  FROM aigenerateexp.bronze.v_contracts
);

Notice the WITH SCHEMA uses INT for seat count, DECIMAL for monetary values, and VARCHAR for text fields. The LLM converts the free-text values to the types you specify. If a contract note says "75 seats," the seat_count column returns the integer 75.

How WITH SCHEMA Changes the Output

Without WITH SCHEMA, AI_GENERATE returns a VARCHAR with the LLM's freeform response. This is harder to work with downstream:

-- Without WITH SCHEMA: returns plain text
SELECT AI_GENERATE(
  'Extract the sender name and company from this email',
  email_body
) AS raw_text
FROM aigenerateexp.bronze.v_emails
LIMIT 3;

The raw text might look like: "Sender: Sarah Chen, Company: Acme Corp" but there's no guarantee of consistent formatting across rows. With WITH SCHEMA, every row returns the same column structure, making the output predictable and queryable.

Persisting Results with CTAS

Materialize your extracted data into Iceberg tables to avoid repeated LLM calls:

CREATE TABLE aigenerateexp.gold.emails_extracted AS
SELECT * FROM aigenerateexp.gold.v_email_extracted;

CREATE TABLE aigenerateexp.gold.contracts_extracted AS
SELECT * FROM aigenerateexp.gold.v_contract_details;

Once materialized, you can run standard SQL analytics on the extracted fields without incurring LLM token costs. Refresh the tables when new emails or contracts arrive.

Step 6: Enable AI-Generated Wikis and Tags

Click Admin in the left sidebar, then go to Project Settings.
Select the Preferences tab.
Scroll to the AI section and enable Generate Wikis and Labels.
Go to the Catalog and navigate to your Gold views under aigenerateexp.gold.
Click the Edit button (pencil icon) next to each view.
In the Details tab, click Generate Wiki and Generate Tags.
Repeat for all Gold views.

Enhance the generated wikis with context like "sender_name and company are LLM-extracted from raw email text. Urgency is classified by the LLM based on language cues like 'urgent', 'critical', and 'immediate'."

Step 7: Ask Questions with the AI Agent

"Which companies sent urgent emails?"

The Agent queries v_email_extracted, filters by urgency containing 'urgent' or 'critical', and returns the company names and topics.

"Show me a chart of email topics by urgency level"

The Agent groups by topic and urgency in v_email_extracted and creates a visualization showing which topics generate the most urgent communications.

"List all won deals over $100,000 with their key contacts"

The Agent filters v_contract_details for deal_status = 'Won' and total_contract_value > 100000, returning client names, products, values, and key contacts.

"Create a chart showing total contract value by account manager and deal status"

The Agent creates a stacked bar chart from v_contract_details comparing each account manager's total pipeline across Won, Lost, Pending, and Expansion statuses.

Processing Unstructured Files with AI_GENERATE and LIST_FILES

The examples above process text that's already stored in table columns. But many organizations have unstructured files, such as PDFs, text documents, images, and scanned invoices, sitting in object storage (S3, Azure Blob, GCS) that have never been queryable through SQL.

Dremio's LIST_FILES table function bridges this gap. It recursively lists files from a connected source and returns metadata about each file. Combined with AI_GENERATE, you can read file content and extract structured data from documents that were previously invisible to your analytics platform.

How LIST_FILES Works

LIST_FILES is a table function that returns metadata for files in a connected storage source:

SELECT *
FROM TABLE(
  LIST_FILES(
    path => 'your_s3_source.folder_name',
    recursive => true
  )
);

The function returns columns including the file source, path, size, and last modification time. This metadata feeds into AI_GENERATE as file references.

Hypothetical Example: Invoice Processing

Suppose you have an S3 bucket connected to Dremio as a source called company_s3, with a folder /invoices/2025/ containing PDF invoices from vendors. Here's how you'd extract structured data:

-- Step 1: List all invoice files
SELECT *
FROM TABLE(
  LIST_FILES(
    path => 'company_s3.invoices.2025',
    recursive => true
  )
);

-- Step 2: Extract structured data from each invoice
SELECT
  invoice_data.vendor_name,
  invoice_data.invoice_number,
  invoice_data.invoice_date,
  invoice_data.total_amount,
  invoice_data.currency,
  invoice_data.line_items
FROM (
  SELECT AI_GENERATE(
    'Extract the vendor name, invoice number, date, total amount, currency, and a summary of line items from this invoice.',
    file_content
    WITH SCHEMA (
      vendor_name VARCHAR,
      invoice_number VARCHAR,
      invoice_date VARCHAR,
      total_amount DECIMAL(12,2),
      currency VARCHAR,
      line_items VARCHAR
    )
  ) AS invoice_data
  FROM TABLE(
    LIST_FILES(
      path => 'company_s3.invoices.2025',
      recursive => true
    )
  )
);

Hypothetical Example: Resume Screening

An HR team stores candidate resumes as PDFs in an S3 bucket. AI_GENERATE extracts candidate information for structured analysis:

SELECT
  candidate.full_name,
  candidate.email,
  candidate.years_experience,
  candidate.primary_skill,
  candidate.education_level,
  candidate.current_company
FROM (
  SELECT AI_GENERATE(
    'Extract candidate information from this resume. For years_experience provide a numeric estimate.',
    file_content
    WITH SCHEMA (
      full_name VARCHAR,
      email VARCHAR,
      years_experience INT,
      primary_skill VARCHAR,
      education_level VARCHAR,
      current_company VARCHAR
    )
  ) AS candidate
  FROM TABLE(
    LIST_FILES(
      path => 'hr_s3.resumes.2025_q4',
      recursive => true
    )
  )
);

Hypothetical Example: Quarterly Report Analysis

Finance stores quarterly PDF reports from subsidiaries. Extract key financial metrics without manual reading:

SELECT
  metrics.subsidiary_name,
  metrics.quarter,
  metrics.total_revenue,
  metrics.net_income,
  metrics.headcount,
  metrics.key_risks
FROM (
  SELECT AI_GENERATE(
    'Extract financial summary data from this quarterly report.',
    file_content
    WITH SCHEMA (
      subsidiary_name VARCHAR,
      quarter VARCHAR,
      total_revenue DECIMAL(15,2),
      net_income DECIMAL(15,2),
      headcount INT,
      key_risks VARCHAR
    )
  ) AS metrics
  FROM TABLE(
    LIST_FILES(
      path => 'finance_s3.quarterly_reports.2025',
      recursive => true
    )
  )
);

Materializing File Extraction Results

Once you've extracted structured data from files, persist it as an Iceberg table:

CREATE TABLE aigenerateexp.gold.invoices_extracted AS
SELECT
  invoice_data.vendor_name,
  invoice_data.invoice_number,
  invoice_data.total_amount,
  invoice_data.currency
FROM (
  SELECT AI_GENERATE(
    'Extract invoice details',
    file_content
    WITH SCHEMA (vendor_name VARCHAR, invoice_number VARCHAR, total_amount DECIMAL(12,2), currency VARCHAR)
  ) AS invoice_data
  FROM TABLE(LIST_FILES(path => 'company_s3.invoices.2025', recursive => true))
);

This creates a governed, queryable Iceberg table from raw PDF invoices. The table supports time travel, schema evolution, and ACID transactions. Build Reflections on it for dashboard acceleration.

Key Considerations for LIST_FILES + AI_GENERATE

Source connectivity: LIST_FILES requires a connected storage source (S3, Azure Storage, GCS) in your Dremio project. The source must be configured with appropriate read permissions.

File format support: Dremio's AI functions can process text-based content including PDFs, text files, and document formats. The LLM interprets the file content and extracts fields per your schema definition.

Token costs: Processing files through the LLM consumes tokens proportional to file size. Filter your LIST_FILES results before passing them to AI_GENERATE to avoid processing unnecessary files:

-- Filter to only recent files before AI processing
SELECT AI_GENERATE(...)
FROM TABLE(LIST_FILES(path => 'company_s3.invoices.2025', recursive => true))
WHERE modification_time > TIMESTAMP '2025-09-01 00:00:00';

Engine routing: Use query_calls_ai_functions() to route file processing queries to a dedicated engine, isolating heavy batch extraction from your regular analytical workloads.

Why Apache Iceberg Matters

Extracted data stored as Iceberg tables benefits from automated performance management. As your extraction pipeline grows from hundreds to thousands of documents, Iceberg's compaction, manifest optimization, and clustering keep query performance consistent without manual tuning.

Iceberg vs. Federated for AI_GENERATE Workloads

Use CTAS materialization when: You're extracting from historical documents (past invoices, old contracts, archived emails). Run the extraction once, query the results forever.

Use live views when: You need real-time extraction from a continuously updating text column in a federated database. Pair with manual Reflections to cache results at a controlled refresh interval, balancing extraction cost against data freshness.

Next Steps

Connect your real data sources — replace simulated tables with federated connections to your email system, CRM, and document storage
Connect an S3 or Azure source — enable LIST_FILES processing on your actual unstructured file repositories
Add FGAC — mask extracted PII fields (emails, phone numbers, names) for downstream consumers who shouldn't see personal data
Build Reflections — create Reflections on CTAS-materialized extraction tables for fast dashboard queries at zero LLM cost

If your organization has unstructured text trapped in database columns or files sitting unanalyzed in object storage, AI_GENERATE turns that text into structured, queryable, governed data. Define a schema, write a prompt, and run a query. The extraction happens inside your lakehouse with the same access controls and governance that apply to all your other data.

Try Dremio Cloud free for 30 days and start extracting structured data from your unstructured text.

Connect Oracle Database to Dremio Cloud: Enterprise Analytics Without Data Movement

Alex Merced — Sun, 01 Mar 2026 11:00:00 GMT

Oracle Database runs the most critical enterprise applications in the world — ERP systems, financial ledgers, supply chain management, and HR platforms. These systems generate massive volumes of data that business teams want to analyze, but running analytical queries directly against Oracle is expensive (license costs scale with CPU usage), complex (Oracle-specific SQL dialects and tooling), and risky (heavy queries can impact transactional performance).

Dremio Cloud connects to Oracle Database and queries it in place using standard SQL. You don't need to license additional Oracle tools, build ETL pipelines, or export data to a separate warehouse. Dremio pushes filters and aggregations to Oracle, fetches only the results, and lets you join Oracle data with every other source in your organization in a single query.

This guide walks through the complete setup, including Oracle-specific features like native encryption, user impersonation, service name configuration, and the extensive predicate pushdown support.

Why Oracle Users Need Dremio

Oracle licensing costs make analytics expensive. Oracle licenses are typically tied to CPU cores. Running analytical workloads on your production Oracle instance consumes CPU, which means higher licensing costs. Dremio's Reflections create pre-computed copies of frequently queried Oracle data. After the initial query, subsequent analytics hit the Reflection — not Oracle — reducing CPU consumption and license exposure.

Cross-system analytics require ETL. Your financial data is in Oracle, your CRM data is in PostgreSQL, and your marketing data is in S3. Without a federation layer, joining these requires building ETL pipelines that extract data from each source, transform it, and load it into a central warehouse. That's months of engineering work. Dremio federates across all three sources with a single SQL query.

Oracle's analytical tooling is Oracle-specific. Oracle Analytics Cloud, Oracle BI, and Oracle Data Integrator work well within the Oracle ecosystem but don't extend to non-Oracle data. Dremio provides a vendor-neutral SQL layer that works with any BI tool (Tableau, Power BI, Looker) via Arrow Flight or ODBC, covering Oracle and every other connected source.

No semantic layer for AI. Oracle tables use technical names and lack the business context that AI agents need to generate accurate SQL. Dremio's semantic layer lets you create views with business logic, attach wiki descriptions, and enable the AI Agent to answer questions like "What's our quarterly revenue by product line?" by understanding what "quarterly revenue" means from your metadata.

Prerequisites

Before connecting Oracle to Dremio Cloud, confirm you have:

Oracle hostname or IP address
Port number — Oracle defaults to 1521
Service name — the Oracle service name (not the SID) for your database
Username and password — an Oracle user with SELECT privileges on the relevant schemas
Network access — port 1521 must be reachable from Dremio Cloud
Dremio Cloud account — sign up free for 30 days

Step-by-Step: Connect Oracle to Dremio Cloud

1. Add the Oracle Source

In the Dremio console, click the "+" in the left sidebar and select Oracle from the database source types.

2. Configure General Settings

Name: A descriptive identifier (e.g., erp-oracle or finance-oracle).
Host: The Oracle server hostname.
Port: Default 1521.
Service Name: The Oracle service name for your database.
Enable TLS encryption: Toggle this on for encrypted connections over TLS.
Oracle Native Encryption: If you don't use TLS, Oracle supports its own encryption protocol. Options are:
- Accepted (default): Allows both encrypted and unencrypted connections.
- Requested: Prefers encryption but accepts unencrypted if not available.
- Required: Only encrypted connections allowed.
- Rejected: No encryption.

You can use either TLS or Oracle Native Encryption, but not both on the same source.

3. Set Authentication

Three options:

Master Authentication: Username and password entered directly.
Secret Resource URL: Password stored in AWS Secrets Manager, referenced by ARN.
Kerberos: For environments where Oracle is configured with Kerberos authentication.

4. Configure Advanced Options

Oracle has several unique advanced settings:

Setting	What It Does
Use timezone as connection region	Uses the timezone to set the connection region
Include synonyms	Makes Oracle synonyms visible as datasets in Dremio
Map Oracle DATE to TIMESTAMP	Oracle's `DATE` type includes time components. Enable this to expose them as `TIMESTAMP` in Dremio instead of truncating to `DATE`
Record fetch size	Rows per batch (default 200, set 0 for automatic)
Use LDAP Naming Services	Authenticate via LDAP rather than Oracle's local user database
User Impersonation	Run queries under each Dremio user's own Oracle credentials (see below)
Connection Properties	Custom JDBC parameters

5. User Impersonation (Optional but Valuable)

Oracle supports user impersonation through proxy authentication. This means each Dremio user runs queries under their own Oracle username, with their own Oracle permissions, rather than sharing a single service account.

To set this up:

Ensure each Dremio user has a matching username in Oracle.
In Oracle, grant proxy authentication: ALTER USER analyst_user GRANT CONNECT THROUGH dremio_service_user;
In Dremio's source settings, enable User Impersonation under Advanced Options.

This is particularly valuable in regulated industries where audit trails need to track which individual accessed which data.

6. Save the Connection

Configure Reflection Refresh, Metadata Refresh, and Privileges as needed, then click Save.

Query Oracle Data from Dremio

Browse your Oracle schemas and tables, then run standard SQL:

SELECT department_id, department_name, manager_id, location_id
FROM "erp-oracle".HR.DEPARTMENTS
WHERE location_id = 1700;

Dremio pushes the WHERE clause to Oracle and transfers only the matching rows.

Federate Oracle with Other Sources

Combine Oracle ERP data with S3 data and PostgreSQL data in one query:

SELECT
  d.department_name,
  COUNT(e.employee_id) AS headcount,
  AVG(e.salary) AS avg_salary,
  SUM(b.budget_amount) AS total_budget
FROM "erp-oracle".HR.DEPARTMENTS d
JOIN "erp-oracle".HR.EMPLOYEES e ON d.department_id = e.department_id
LEFT JOIN "finance-postgres".budgets.dept_budgets b ON d.department_id = b.dept_id
GROUP BY d.department_name
ORDER BY total_budget DESC;

Oracle handles the department-employee join (predicate pushdown), and Dremio handles the cross-source join with PostgreSQL budget data.

Predicate Pushdown Support

Oracle has one of the most comprehensive pushdown profiles in Dremio. The engine offloads:

All standard comparisons and logical operators
Aggregations: SUM, AVG, COUNT, MIN, MAX, STDDEV, MEDIAN, VAR_POP, VAR_SAMP, COVAR_POP, COVAR_SAMP, PERCENTILE_CONT, PERCENTILE_DISC
Math functions: ABS, CEIL, FLOOR, ROUND, MOD, SQRT, POWER, LOG, EXP, SIGN, trigonometric functions (SIN, COS, TAN, ASIN, ACOS, ATAN, SINH, COSH, TANH)
String functions: CONCAT, SUBSTR, LENGTH, LOWER, UPPER, TRIM, REPLACE, REVERSE, LPAD, RPAD
Date functions: DATE_ADD, DATE_SUB, DATE_TRUNC, EXTRACT, ADD_MONTHS, LAST_DAY, TO_CHAR, TO_DATE, TRUNC

This extensive pushdown support means Oracle does most of the heavy lifting for filtering and aggregation, and Dremio only transfers the summarized results across the network.

Data Type Mapping

Oracle	Dremio	Notes
NUMBER	DECIMAL	Preserves precision
VARCHAR2 / NVARCHAR2 / CHAR / NCHAR	VARCHAR
DATE	DATE or TIMESTAMP	Use advanced option to map to TIMESTAMP
TIMESTAMP	TIMESTAMP
BINARY_FLOAT	FLOAT
BINARY_DOUBLE	DOUBLE
FLOAT	DOUBLE
BLOB / RAW / LONG RAW	VARBINARY
LONG	VARCHAR
INTERVALDS	INTERVAL (day to seconds)
INTERVALYM	INTERVAL (years to months)

CLOB, XMLTYPE, and Oracle spatial types are not supported through the connector.

Build a Semantic Layer Over Oracle

Create views that translate Oracle's technical schema into business-friendly analytics:

CREATE VIEW analytics.gold.department_performance AS
SELECT
  d.department_name,
  COUNT(e.employee_id) AS employee_count,
  ROUND(AVG(e.salary), 2) AS avg_salary,
  MAX(e.hire_date) AS most_recent_hire,
  CASE
    WHEN COUNT(e.employee_id) > 50 THEN 'Large'
    WHEN COUNT(e.employee_id) > 20 THEN 'Medium'
    ELSE 'Small'
  END AS department_size
FROM "erp-oracle".HR.DEPARTMENTS d
LEFT JOIN "erp-oracle".HR.EMPLOYEES e ON d.department_id = e.department_id
GROUP BY d.department_name;

Attach wiki context via the Catalog (edit pencil icon → Details → Generate Wiki/Tags) so the AI Agent can answer questions like "Which large departments have the highest average salary?"

When to Keep Data in Oracle vs. Migrate to Iceberg

Keep in Oracle: Actively transactional data (current orders, inventory, ledger entries), data that applications read and write frequently, data subject to Oracle-specific constraints and triggers.

Migrate to Iceberg: Historical archives (closed fiscal quarters, past-year orders), aggregated reporting tables, datasets queried heavily for analytics but rarely written.

For data that stays in Oracle, create manual Reflections with a refresh schedule that balances data freshness against Oracle CPU usage. For migrated data, Dremio's Open Catalog provides automated compaction, time travel, and Autonomous Reflections.

AI-Powered Analytics on Oracle Data

Dremio AI Agent

The AI Agent lets business users ask questions about Oracle data in plain English. An HR director asks "Which large departments have the highest average salary?" and the Agent generates accurate SQL by reading the wiki descriptions on your department_performance view. The Agent understands what "large" means (employee_count > 50) because you've defined it in the semantic layer.

This is particularly valuable for Oracle environments where decades of institutional knowledge about schema structures, table naming conventions (like HR.DEPARTMENTS), and column semantics lives in senior DBAs' heads.

Dremio MCP Server

The Dremio MCP Server connects Claude, ChatGPT, and other AI clients to your Oracle data through Dremio:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs (e.g., https://claude.ai/api/mcp/auth_callback for Claude, https://chatgpt.com/connector_platform_oauth_redirect for ChatGPT)
Connect using mcp.dremio.cloud/mcp/{project_id} (US) or mcp.eu.dremio.cloud/mcp/{project_id} (EU)

A CFO asks Claude "Compare department headcount and budget utilization across our Oracle ERP" and gets governed, accurate results from Oracle data — without knowing SQL or Oracle table structures.

AI SQL Functions

Use AI directly in queries against Oracle data:

-- Classify departments by operational health
SELECT
  department_name,
  employee_count,
  avg_salary,
  department_size,
  AI_CLASSIFY(
    'Based on these HR metrics, classify the department health',
    'Department: ' || department_name || ', Employees: ' || CAST(employee_count AS VARCHAR) || ', Avg Salary: $' || CAST(avg_salary AS VARCHAR) || ', Size: ' || department_size,
    ARRAY['Thriving', 'Stable', 'Understaffed', 'Needs Attention']
  ) AS department_health
FROM analytics.gold.department_performance;

-- Generate executive briefings from Oracle data
SELECT
  department_name,
  AI_GENERATE(
    'Write a one-sentence executive summary for this department',
    'Department: ' || department_name || ', Headcount: ' || CAST(employee_count AS VARCHAR) || ', Avg Salary: $' || CAST(avg_salary AS VARCHAR) || ', Most Recent Hire: ' || CAST(most_recent_hire AS VARCHAR)
  ) AS executive_summary
FROM analytics.gold.department_performance
WHERE department_size = 'Large';

AI_CLASSIFY runs LLM inference to categorize departments. AI_GENERATE creates narrative summaries. Both run inline in your SQL queries, enriching Oracle data with AI.

Reflections for Performance

Oracle Database licensing is expensive — especially Enterprise Edition with Analytics and Diagnostics Packs. Reflections offload analytical queries:

Navigate to the view in the Catalog
Click the Reflections tab
Choose Raw Reflection or Aggregation Reflection
Select columns and set the Refresh Interval — for HR data, daily; for financial data, match to reporting cycles
Click Save

BI tools get sub-second responses from Reflections. Oracle focuses on transactional workloads. A department performance dashboard with hourly refreshes generates zero Oracle CPU consumption after the Reflection is built.

Governance on Oracle Data

Oracle has its own security model (Oracle Database Vault, VPD), but it doesn't extend to non-Oracle sources. Dremio's Fine-Grained Access Control (FGAC) provides unified governance:

Column masking: Mask salary data, employee SSNs, and performance ratings from specific roles. An HR generalist sees headcount but not compensation details.
Row-level filtering: Department-level access — a department manager sees only their department. Regional HR sees only their region.
Unified policies: Same governance applies across Oracle, PostgreSQL, S3, and all other sources.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Tableau: Dremio connector for direct Arrow Flight access
Power BI: Dremio ODBC driver or native connector — no Oracle client needed
Python/Pandas: pyarrow.flight for programmatic data access
Looker: Connect via JDBC
dbt: dbt-dremio for SQL-based transformations

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query Oracle data from their IDE. Ask Copilot "Show me understaffed departments from Oracle HR" and get SQL generated from your semantic layer — without knowing Oracle schema conventions.

When to Keep Data in Oracle vs. Migrate

Keep in Oracle: Transactional data for active applications, data with PL/SQL dependencies (stored procedures, triggers, packages), data subject to Oracle RAC clustering, data managed by Oracle GoldenGate replication.

Migrate to Iceberg: Historical HR data and archives, closed fiscal year financials, data consumed by non-Oracle tools, datasets where Oracle per-core licensing exceeds analytical value. Migrated Iceberg tables get automatic compaction, time travel, and Autonomous Reflections.

For data staying in Oracle, create manual Reflections to reduce Oracle CPU load. For migrated Iceberg data, Dremio handles optimization automatically.

Get Started

Oracle Database users pay a premium for their database's reliability and enterprise features. Dremio Cloud lets you extract analytical value from that data without additional Oracle licensing, ETL pipelines, or vendor-specific BI tools.

Try Dremio Cloud free for 30 days and connect your Oracle databases alongside your other data sources.

Generate Summaries and Insights with Dremio's AI_COMPLETE Function

Alex Merced — Sun, 01 Mar 2026 10:00:00 GMT

Every data team has a version of this problem: a table full of raw data that needs human-readable summaries, translations, or narrative descriptions. Product descriptions that need rewriting for a new market. Customer records that need one-sentence executive summaries. Support interactions that need post-call notes.

AI_COMPLETE brings an LLM directly into your SQL query to produce that text. You write a prompt, pass in your data columns, and get generated text back as a VARCHAR. No Python notebooks, no external APIs, no data exports.

This tutorial builds a complete product analytics pipeline in a fresh Dremio Cloud account. You'll create sample product and sales data, build a medallion architecture, and use AI_COMPLETE to generate product summaries, executive briefings, marketing copy, and translations, all inside SQL.

What You'll Build

By the end of this tutorial, you'll have:

A product catalog with 50+ products and 50+ sales records
Bronze views that standardize raw data
Silver views that compute sales metrics per product
Gold views that use AI_COMPLETE to generate summaries, marketing descriptions, and translated content
Materialized Iceberg tables that persist generated text for dashboards
Wiki metadata that enables the AI Agent to answer natural language questions about your enriched catalog

Prerequisites

Dremio Cloud account — sign up free for 30 days with $400 in compute credits
AI enabled — go to Admin → Project Settings → Preferences → AI section and enable AI features
Model Provider configured — Dremio provides a hosted LLM by default, or connect your own (OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI)

Note: Tables in the built-in Open Catalog use folder.subfolder.table_name without a catalog prefix. External sources use source_name.schema.table_name.

Understanding AI_COMPLETE

AI_COMPLETE sends a prompt to your configured LLM and returns the generated text as a VARCHAR. The function signature:

AI_COMPLETE(
  [model_name VARCHAR,]
  prompt VARCHAR
) → VARCHAR

Parameters:

model_name (optional) — specify a model like 'openai.gpt-4o'. Format is modelProvider.modelName. If omitted, Dremio uses your default model.
prompt — the text instruction for the LLM. Typically you concatenate a task description with column values to give the model both the instruction and the data context.

The key difference from AI_CLASSIFY is that AI_COMPLETE returns free-text output. There's no array of allowed values. The LLM generates whatever text the prompt asks for: a summary, a translation, a paragraph, a sentence, or a structured response.

This flexibility is both the strength and the risk. A well-crafted prompt produces consistent, useful output. A vague prompt produces inconsistent results. Prompt engineering matters here more than with classification.

Step 1: Create Your Folder Structure

Open the SQL Runner from the left sidebar in Dremio Cloud:

CREATE FOLDER IF NOT EXISTS aicompleteexp;
CREATE FOLDER IF NOT EXISTS aicompleteexp.catalog_data;
CREATE FOLDER IF NOT EXISTS aicompleteexp.bronze;
CREATE FOLDER IF NOT EXISTS aicompleteexp.silver;
CREATE FOLDER IF NOT EXISTS aicompleteexp.gold;

Step 2: Seed Your Sample Data

Products Table

This table simulates a SaaS product catalog with technical descriptions, pricing tiers, and categories. These descriptions are the raw material that AI_COMPLETE will use to generate marketing copy and summaries.

CREATE TABLE aicompleteexp.catalog_data.products (
  product_id INT,
  product_name VARCHAR,
  category VARCHAR,
  description VARCHAR,
  price_monthly DECIMAL(10,2),
  launch_date DATE,
  target_audience VARCHAR
);

INSERT INTO aicompleteexp.catalog_data.products VALUES
(1, 'CloudSync Pro', 'Storage', 'Enterprise file synchronization platform supporting real-time sync across Windows Mac and Linux with conflict resolution selective sync and 256-bit AES encryption at rest and in transit', 29.99, '2023-03-15', 'IT Teams'),
(2, 'DataVault Enterprise', 'Security', 'Zero-knowledge encrypted cloud storage with SOC 2 Type II certification automated backup deduplication granular access controls and 99.99% uptime SLA for regulated industries', 89.99, '2022-11-01', 'Compliance Officers'),
(3, 'QuickReport', 'Analytics', 'Self-service business intelligence tool with drag-and-drop report builder 50+ chart types scheduled email delivery PDF export and REST API for automated report generation', 49.99, '2024-01-20', 'Business Analysts'),
(4, 'DevPipeline', 'DevOps', 'CI/CD platform with parallel build execution Docker and Kubernetes native deployment auto-scaling runners built-in secret management and integration with GitHub GitLab and Bitbucket', 79.99, '2023-07-10', 'Engineering Teams'),
(5, 'MailForge', 'Marketing', 'Email marketing automation platform with AI-powered subject line optimization A/B testing dynamic content personalization and real-time deliverability monitoring across 50+ ISPs', 39.99, '2024-05-01', 'Marketing Teams'),
(6, 'HelpDesk360', 'Support', 'Omnichannel customer support platform supporting email chat phone and social media with SLA tracking auto-routing knowledge base integration and customer satisfaction scoring', 59.99, '2023-09-15', 'Support Managers'),
(7, 'FormBuilder', 'Productivity', 'No-code form and survey creation tool with conditional logic payment collection 200+ templates analytics dashboard and WCAG 2.1 AA accessibility compliance', 19.99, '2024-02-28', 'Operations Teams'),
(8, 'APIGateway Pro', 'Infrastructure', 'API management platform with rate limiting OAuth 2.0 authentication request transformation caching analytics dashboard and support for REST GraphQL and gRPC protocols', 99.99, '2023-01-05', 'Platform Engineers'),
(9, 'InventoryTrack', 'Commerce', 'Multi-warehouse inventory management system with barcode scanning lot tracking reorder alerts multi-currency support and integration with Shopify WooCommerce and Amazon', 44.99, '2024-04-10', 'E-commerce Managers'),
(10, 'TeamBoard', 'Collaboration', 'Visual project management platform with Kanban Gantt and timeline views time tracking resource allocation dependencies and Slack Microsoft Teams integration', 24.99, '2023-06-20', 'Project Managers'),
(11, 'SecureSign', 'Legal', 'Electronic signature platform with legally binding signatures audit trails multi-party signing workflows template library and compliance with eIDAS UETA and ESIGN regulations', 34.99, '2024-03-01', 'Legal Teams'),
(12, 'DataStream', 'Data', 'Real-time data pipeline platform supporting Kafka Pulsar and Kinesis with schema registry exactly-once processing dead letter queues and built-in data quality checks', 149.99, '2023-04-18', 'Data Engineers'),
(13, 'AdOptimizer', 'Marketing', 'Cross-channel advertising platform with automated bid management audience segmentation attribution modeling creative testing and budget pacing across Google Facebook and LinkedIn', 199.99, '2024-06-15', 'Performance Marketers'),
(14, 'ContractManager', 'Legal', 'Contract lifecycle management platform with AI-assisted clause extraction version tracking approval workflows obligation monitoring and integration with Salesforce and HubSpot', 69.99, '2023-08-22', 'Legal Operations'),
(15, 'LogInsight', 'Infrastructure', 'Log aggregation and analysis platform with full-text search pattern detection anomaly alerts custom dashboards and retention policies supporting up to 10TB daily ingestion', 119.99, '2023-02-14', 'SRE Teams'),
(16, 'PayFlow', 'Finance', 'Payment processing platform with support for 135 currencies PCI DSS Level 1 compliance recurring billing invoice generation and fraud detection using ML models', 0.00, '2024-01-10', 'Finance Teams'),
(17, 'ChatAssist', 'Support', 'AI-powered chatbot platform with natural language understanding intent classification handoff to human agents conversation analytics and multi-language support for 40+ languages', 74.99, '2024-07-01', 'Customer Experience'),
(18, 'SchedulePro', 'HR', 'Employee scheduling platform with shift management availability tracking overtime calculation labor cost forecasting and integration with ADP Workday and BambooHR payroll systems', 14.99, '2023-11-05', 'HR Managers'),
(19, 'CloudBackup', 'Storage', 'Automated cloud backup solution with incremental backups point-in-time recovery cross-region replication ransomware protection and support for AWS Azure and GCP workloads', 54.99, '2023-05-30', 'IT Administrators'),
(20, 'DesignHub', 'Productivity', 'Collaborative design platform with real-time co-editing version history component libraries handoff-to-dev specs and integration with Figma Sketch and Adobe XD import', 29.99, '2024-08-10', 'Design Teams');

Sales Data Table

This table tracks monthly sales performance for each product, giving us the raw numbers that AI_COMPLETE will summarize into narrative insights.

CREATE TABLE aicompleteexp.catalog_data.sales_data (
  sale_id INT,
  product_id INT,
  month_year VARCHAR,
  units_sold INT,
  revenue DECIMAL(12,2),
  new_customers INT,
  churned_customers INT,
  region VARCHAR
);

INSERT INTO aicompleteexp.catalog_data.sales_data VALUES
(1, 1, '2025-07', 342, 10256.58, 89, 12, 'North America'),
(2, 1, '2025-08', 389, 11666.11, 102, 8, 'North America'),
(3, 1, '2025-09', 415, 12446.85, 95, 15, 'Europe'),
(4, 2, '2025-07', 156, 14037.44, 34, 5, 'North America'),
(5, 2, '2025-08', 178, 16017.22, 41, 3, 'Europe'),
(6, 2, '2025-09', 201, 18087.99, 52, 7, 'North America'),
(7, 3, '2025-07', 267, 13346.33, 73, 18, 'North America'),
(8, 3, '2025-08', 234, 11697.66, 58, 22, 'Europe'),
(9, 3, '2025-09', 298, 14894.02, 81, 14, 'Asia Pacific'),
(10, 4, '2025-07', 123, 9837.77, 28, 4, 'North America'),
(11, 4, '2025-08', 145, 11598.55, 35, 6, 'Europe'),
(12, 4, '2025-09', 167, 13358.33, 42, 3, 'North America'),
(13, 5, '2025-07', 445, 17795.55, 112, 25, 'North America'),
(14, 5, '2025-08', 478, 19115.22, 98, 30, 'Europe'),
(15, 5, '2025-09', 512, 20475.88, 134, 19, 'Asia Pacific'),
(16, 6, '2025-07', 198, 11877.02, 45, 11, 'North America'),
(17, 6, '2025-08', 212, 12717.88, 52, 8, 'Europe'),
(18, 6, '2025-09', 189, 11331.11, 38, 16, 'North America'),
(19, 7, '2025-07', 567, 11334.33, 145, 32, 'North America'),
(20, 7, '2025-08', 612, 12234.88, 160, 28, 'Europe'),
(21, 7, '2025-09', 589, 11774.11, 138, 35, 'Asia Pacific'),
(22, 8, '2025-07', 89, 8899.11, 15, 2, 'North America'),
(23, 8, '2025-08', 95, 9499.05, 18, 1, 'Europe'),
(24, 8, '2025-09', 102, 10198.98, 22, 3, 'North America'),
(25, 9, '2025-07', 312, 14035.88, 78, 14, 'North America'),
(26, 9, '2025-08', 287, 12911.13, 65, 19, 'Europe'),
(27, 9, '2025-09', 345, 15520.55, 92, 11, 'Asia Pacific'),
(28, 10, '2025-07', 234, 5847.66, 67, 20, 'North America'),
(29, 10, '2025-08', 256, 6397.44, 72, 15, 'Europe'),
(30, 10, '2025-09', 278, 6946.22, 80, 18, 'Asia Pacific'),
(31, 11, '2025-07', 189, 6613.11, 48, 9, 'North America'),
(32, 11, '2025-08', 201, 7032.99, 55, 7, 'Europe'),
(33, 11, '2025-09', 223, 7802.77, 62, 5, 'North America'),
(34, 12, '2025-07', 67, 10049.33, 12, 1, 'North America'),
(35, 12, '2025-08', 78, 11699.22, 16, 2, 'Europe'),
(36, 12, '2025-09', 82, 12299.18, 19, 1, 'North America'),
(37, 13, '2025-07', 134, 26793.66, 28, 6, 'North America'),
(38, 13, '2025-08', 145, 28993.55, 32, 4, 'Europe'),
(39, 13, '2025-09', 167, 33393.33, 41, 8, 'North America'),
(40, 14, '2025-07', 112, 7838.88, 25, 5, 'North America'),
(41, 14, '2025-08', 128, 8959.72, 30, 3, 'Europe'),
(42, 14, '2025-09', 145, 10149.55, 38, 4, 'North America'),
(43, 15, '2025-07', 56, 6719.44, 10, 2, 'North America'),
(44, 15, '2025-08', 62, 7439.38, 13, 1, 'Europe'),
(45, 15, '2025-09', 71, 8519.29, 17, 2, 'North America'),
(46, 16, '2025-07', 890, 0.00, 234, 45, 'North America'),
(47, 16, '2025-08', 1023, 0.00, 267, 38, 'Europe'),
(48, 16, '2025-09', 1156, 0.00, 301, 52, 'Asia Pacific'),
(49, 17, '2025-07', 145, 10873.55, 38, 8, 'North America'),
(50, 17, '2025-08', 167, 12522.33, 45, 6, 'Europe'),
(51, 17, '2025-09', 189, 14173.11, 52, 10, 'Asia Pacific'),
(52, 18, '2025-07', 423, 6341.77, 110, 28, 'North America'),
(53, 18, '2025-08', 456, 6836.44, 118, 22, 'Europe'),
(54, 18, '2025-09', 489, 7331.11, 125, 30, 'Asia Pacific');

Step 3: Build Bronze Views

Bronze views standardize column names and cast dates to timestamps. No business logic at this layer.

CREATE OR REPLACE VIEW aicompleteexp.bronze.v_products AS
SELECT
  product_id,
  product_name,
  category,
  description,
  price_monthly,
  CAST(launch_date AS TIMESTAMP) AS launch_timestamp,
  target_audience
FROM aicompleteexp.catalog_data.products;

CREATE OR REPLACE VIEW aicompleteexp.bronze.v_sales AS
SELECT
  sale_id,
  product_id,
  month_year,
  units_sold,
  revenue,
  new_customers,
  churned_customers,
  region
FROM aicompleteexp.catalog_data.sales_data;

Step 4: Build Silver Views

This Silver view aggregates sales performance per product across all months, giving us total revenue, total units, average deal size, net customer growth, and growth rate. The AI_COMPLETE function will use these metrics to generate narrative summaries.

CREATE OR REPLACE VIEW aicompleteexp.silver.v_product_performance AS
SELECT
  p.product_id,
  p.product_name,
  p.category,
  p.description,
  p.price_monthly,
  p.target_audience,
  COALESCE(SUM(s.units_sold), 0) AS total_units,
  COALESCE(SUM(s.revenue), 0) AS total_revenue,
  COALESCE(SUM(s.new_customers), 0) AS total_new_customers,
  COALESCE(SUM(s.churned_customers), 0) AS total_churned,
  COALESCE(SUM(s.new_customers), 0) - COALESCE(SUM(s.churned_customers), 0) AS net_customer_growth,
  CASE
    WHEN COALESCE(SUM(s.units_sold), 0) > 0
    THEN ROUND(COALESCE(SUM(s.revenue), 0) / SUM(s.units_sold), 2)
    ELSE 0
  END AS avg_revenue_per_unit,
  COUNT(DISTINCT s.region) AS regions_active
FROM aicompleteexp.bronze.v_products p
LEFT JOIN aicompleteexp.bronze.v_sales s ON p.product_id = s.product_id
GROUP BY p.product_id, p.product_name, p.category, p.description, p.price_monthly, p.target_audience;

Step 5: Build Gold Views with AI_COMPLETE

Gold View 1: Executive Product Summaries

This view generates a one-sentence executive summary for each product based on its sales performance. Product managers use these summaries in weekly reports without manually writing them.

The prompt includes specific data points (revenue, units, customer growth) so the LLM produces factual summaries rather than generic descriptions.

CREATE OR REPLACE VIEW aicompleteexp.gold.v_product_summaries AS
SELECT
  product_id,
  product_name,
  category,
  total_revenue,
  total_units,
  net_customer_growth,
  AI_COMPLETE(
    'Write a single-sentence executive summary for this product. Be specific with numbers. Product: '
    || product_name
    || '. Category: ' || category
    || '. Total revenue: $' || CAST(total_revenue AS VARCHAR)
    || '. Units sold: ' || CAST(total_units AS VARCHAR)
    || '. Net customer growth: ' || CAST(net_customer_growth AS VARCHAR)
    || '. Target audience: ' || target_audience
  ) AS executive_summary
FROM aicompleteexp.silver.v_product_performance;

Gold View 2: Marketing Description Generator

This view transforms technical product descriptions into customer-facing marketing copy. The LLM rewrites the description in a style that emphasizes benefits rather than features, suitable for a product landing page.

CREATE OR REPLACE VIEW aicompleteexp.gold.v_marketing_copy AS
SELECT
  product_id,
  product_name,
  category,
  description AS technical_description,
  price_monthly,
  target_audience,
  AI_COMPLETE(
    'Rewrite this technical product description as a compelling 2-3 sentence marketing paragraph for a product landing page. Focus on benefits not features. Avoid buzzwords like transformative or revolutionary. Product: '
    || product_name
    || '. Technical description: ' || description
    || '. Price: $' || CAST(price_monthly AS VARCHAR) || '/month'
    || '. Target audience: ' || target_audience
  ) AS marketing_description
FROM aicompleteexp.bronze.v_products;

Gold View 3: Translated Descriptions

AI_COMPLETE handles translation by including the target language in the prompt. This view generates Spanish translations of product descriptions for a localization initiative.

CREATE OR REPLACE VIEW aicompleteexp.gold.v_spanish_catalog AS
SELECT
  product_id,
  product_name,
  description AS english_description,
  AI_COMPLETE(
    'Translate this product description to Spanish. Return only the Spanish text, no explanations: ' || description
  ) AS spanish_description
FROM aicompleteexp.bronze.v_products;

Choosing the Right Model

For summarization tasks, you can specify a model optimized for speed or quality:

-- Use a specific high-quality model for executive summaries
SELECT
  product_name,
  AI_COMPLETE(
    'openai.gpt-4o',
    'Write a brief executive summary: Product ' || product_name
    || ' generated $' || CAST(total_revenue AS VARCHAR) || ' in revenue'
  ) AS summary
FROM aicompleteexp.silver.v_product_performance;

Prompt Engineering Patterns

The quality of AI_COMPLETE output depends heavily on prompt structure. Here are patterns that produce consistent results:

Be specific about format: "Write a single-sentence summary" produces better output than "Summarize this." Specify the expected length and format.

Include constraints: "Avoid buzzwords like transformative or revolutionary" steers the LLM away from generic marketing language. "Return only the Spanish text, no explanations" prevents the model from adding unwanted context.

Provide data context: Concatenate actual numbers into the prompt. "Total revenue: $34,000" gives the LLM facts to work with, reducing hallucination. Never ask the LLM to calculate; provide pre-computed metrics and ask it to narrate them.

Test with LIMIT: Before running AI_COMPLETE on your full dataset, test with LIMIT 5 to check output quality and token costs:

SELECT product_name, AI_COMPLETE('Summarize in one sentence: ' || description)
FROM aicompleteexp.bronze.v_products
LIMIT 5;

Persisting Results with CTAS

LLM calls cost tokens on every execution. For dashboards or reports that display generated summaries, materialize the results:

CREATE TABLE aicompleteexp.gold.product_summaries_materialized AS
SELECT * FROM aicompleteexp.gold.v_product_summaries;

Refresh this table on a schedule (weekly, after each sales data update) to keep summaries current without running LLM calls on every dashboard load.

Step 6: Enable AI-Generated Wikis and Tags

Add metadata context to your Gold views so the AI Agent can answer questions about your enriched catalog:

Click Admin in the left sidebar, then go to Project Settings.
Select the Preferences tab.
Scroll to the AI section and enable Generate Wikis and Labels.
Go to the Catalog and navigate to your Gold views under aicompleteexp.gold.
Click the Edit button (pencil icon) next to each view.
In the Details tab, click Generate Wiki and Generate Tags.
Repeat for all Gold views.

To enhance the generated wiki, copy it into the AI Agent and ask for improvements. For example: "Add context explaining that the executive_summary column is generated by an LLM using actual revenue and customer data, and that summaries are refreshed weekly after the sales data pipeline runs."

Step 7: Ask Questions with the AI Agent

Navigate to the AI Agent and try these prompts:

"Which products have the highest net customer growth?"

The Agent queries v_product_summaries, sorts by net_customer_growth, and returns the top products with their AI-generated summaries.

"Show me a chart of total revenue by product category"

The Agent groups products by category in v_product_performance, sums revenue, and generates a bar chart showing which categories drive the most revenue.

"List all products in the Security category with their marketing descriptions"

The Agent filters v_marketing_copy for category = 'Security' and returns product names alongside the LLM-generated marketing paragraphs.

"Create a chart comparing new customers vs churned customers by product"

The Agent queries v_product_performance and creates a grouped bar chart showing customer acquisition and churn side by side, making it easy to spot products with healthy vs. concerning net growth.

Why Apache Iceberg Matters

Your materialized summary tables are Apache Iceberg tables in the built-in Open Catalog. This means:

Time travel: Compare this week's AI-generated summaries with last week's to see how the narrative changed as sales data evolved
Schema evolution: Add new generated columns (like translations to additional languages) without rewriting existing data
ACID transactions: CTAS jobs write atomically; dashboards never see partial results

Iceberg vs. Federated for AI_COMPLETE Workloads

Keep data federated when: Your source data updates frequently and you want the latest products or sales figures in real-time queries. Use manual Reflections to cache results.

Migrate to Iceberg when: You're generating summaries on historical data or building a curated catalog of marketing copy. CTAS materializes the generated text once, and Iceberg's automated performance management (compaction, manifest optimization) keeps the table fast as it grows.

Next Steps

Connect your real product database — replace simulated tables with federated connections to your actual catalog and CRM
Generate descriptions in multiple languages — create additional Gold views with French, German, or Japanese translations
Add FGAC — mask revenue numbers in generated summaries for roles that shouldn't see financial data
Build Reflections — create Reflections on materialized tables to accelerate dashboard queries at zero LLM cost

If your team spends time manually writing product summaries, translating content, or creating executive briefings from raw data, AI_COMPLETE automates that work inside the same SQL engine where your data already lives. Write a prompt, run a query, and get your generated text in the same governed platform where everything else runs.

Try Dremio Cloud free for 30 days and start generating insights with SQL.

Connect MySQL to Dremio Cloud: Federated Analytics Without ETL

Alex Merced — Sun, 01 Mar 2026 10:00:00 GMT

MySQL runs more web applications, SaaS platforms, and e-commerce backends than any other database. It's fast for transactional reads and writes, but it becomes a bottleneck when your data team needs to run analytical queries, join MySQL data with other sources, or build dashboards that don't compete with application traffic.

Dremio Cloud connects directly to MySQL and queries it in place. Your data stays where it is. Dremio pushes filters (called predicate pushdowns) to MySQL when possible, joins MySQL data with any other connected source, and accelerates repeated queries with pre-computed Reflections so your production database isn't hit by every dashboard refresh.

This guide covers everything from prerequisites to federated queries across MySQL and your other data sources.

Why MySQL Users Need Dremio

Analytics compete with application traffic. MySQL was built for OLTP (Online Transaction Processing) — fast inserts, updates, and single-row lookups. Analytical queries that scan millions of rows, compute aggregations, or join large tables create lock contention and slow down application responses. Dremio's Reflections solve this: after the first query, analytical workloads hit Dremio's pre-computed cache instead of MySQL.

Data is siloed. Your order data is in MySQL, customer engagement data is in MongoDB, and marketing attribution data is in S3. Joining these requires building ETL pipelines that extract, transform, and load data into a central warehouse. Dremio eliminates this by querying each source in place and joining the results in its query engine. One SQL query, multiple sources.

Read replicas are expensive and complex. The common workaround for MySQL analytics is creating a read replica. This adds infrastructure cost, replication lag, and operational complexity. Dremio's Reflections provide the same benefit (offloading analytical reads) without a separate database instance. The query optimizer transparently serves results from Reflections when they match.

No built-in semantic layer. MySQL tables have raw column names and no business context. Dremio lets you create views with business logic (like defining what "active customer" means), attach wiki descriptions and labels to those views, and then let the AI Agent answer questions in plain English based on that context.

Prerequisites

Before connecting MySQL to Dremio Cloud, confirm you have:

MySQL hostname or IP address
Port number — MySQL defaults to 3306
Username and password — a MySQL user with SELECT privileges on the tables you want to query
Network access — port 3306 must be reachable from Dremio Cloud. Open the port in your AWS Security Group, Azure NSG, or firewall configuration
Dremio Cloud account — sign up free for 30 days

Step-by-Step: Connect MySQL to Dremio Cloud

1. Add the MySQL Source

In the Dremio console, click the "+" button in the left sidebar, then select MySQL under database sources. Alternatively, go to Databases and click Add database.

2. Configure General Settings

Name: A descriptive identifier (e.g., ecommerce-mysql). This name appears in SQL queries as the source prefix. Cannot include /, :, [, or ].
Host: Your MySQL server's hostname (e.g., my-rds-instance.abc123.us-east-1.rds.amazonaws.com).
Port: Default 3306.
Database (optional): Specify a single database to connect to, or leave blank to access all databases the user can see.

3. Set Authentication

Two options:

No Authentication: For development instances with no password requirement.
Master Credentials: Enter the MySQL username and password with SELECT permissions on your tables.

4. Configure Advanced Options

Setting	What It Does	Default
Net write timeout (in seconds)	How long to wait for data from MySQL before dropping the connection.	60
Record fetch size	Rows per batch. Set to 0 for automatic.	200
Maximum Idle Connections	Idle connection pool size.	8
Connection idle time (s)	Seconds before idle connections close.	60
Query timeout (s)	Maximum query execution time before cancellation.	None
Properties	Custom JDBC connection key-value pairs.	None

5. Set Reflection and Metadata Refresh

Reflection Refresh: Controls how often Dremio re-queries MySQL to update pre-computed Reflections. For frequently changing data, set to 1-4 hours. For stable data, daily or weekly.
Metadata Refresh: Controls how often Dremio checks for new tables or schema changes. Default is 1 hour for both discovery and details.

6. Set Privileges and Save

Optionally restrict which Dremio users can access this source. Click Save.

Query MySQL Data in Dremio

Once connected, browse your MySQL schemas and tables in the SQL Runner:

SELECT order_id, customer_id, total_amount, order_date, status
FROM "ecommerce-mysql".shop.orders
WHERE order_date >= '2024-06-01'
  AND status = 'completed'
ORDER BY total_amount DESC;

Dremio pushes the date filter, status filter, and sort to MySQL — only the matching rows are transferred.

Federate MySQL with Other Sources

Join MySQL order data with S3 clickstream data and PostgreSQL customer profiles in a single query:

SELECT
  c.customer_name,
  c.region,
  COUNT(o.order_id) AS total_orders,
  SUM(o.total_amount) AS total_revenue,
  COUNT(DISTINCT e.session_id) AS web_sessions
FROM "postgres-crm".public.customers c
LEFT JOIN "ecommerce-mysql".shop.orders o
  ON c.customer_id = o.customer_id
LEFT JOIN "s3-analytics".clickstream.sessions e
  ON c.customer_id = e.user_id
GROUP BY c.customer_name, c.region
ORDER BY total_revenue DESC;

No ETL. No data warehouse loading. Three sources, one query.

Build Views and Enable the AI Agent

Create business-friendly views over MySQL data:

CREATE VIEW analytics.gold.order_summary AS
SELECT
  o.order_id,
  o.customer_id,
  o.total_amount,
  CAST(o.order_date AS TIMESTAMP) AS order_timestamp,
  o.status AS order_status,
  CASE
    WHEN o.total_amount > 500 THEN 'High Value'
    WHEN o.total_amount > 100 THEN 'Medium Value'
    ELSE 'Standard'
  END AS order_tier
FROM "ecommerce-mysql".shop.orders o
WHERE o.status IN ('completed', 'shipped');

Then navigate to the Catalog, click Edit (pencil icon) on the view, go to the Details tab, and click Generate Wiki and Generate Tags. This gives Dremio's AI Agent the context it needs to answer questions like "How many high-value orders shipped last month?"

Predicate Pushdown: What Runs on MySQL

Dremio pushes a wide range of operations directly to MySQL, including:

Logical: AND, OR, NOT
Comparisons: =, !=, <, >, <=, >=, LIKE, NOT LIKE, IS NULL, IS NOT NULL
Aggregations: SUM, AVG, COUNT, MIN, MAX, STDDEV, VAR_POP
Math: ABS, CEIL, FLOOR, ROUND, MOD, SQRT, POWER, LOG, EXP
String: CONCAT, SUBSTR, LENGTH, LOWER, UPPER, TRIM, REPLACE, REVERSE
Date/Time: DATE_ADD, DATE_SUB, DATE_TRUNC, EXTRACT, TIMESTAMPADD, TIMESTAMPDIFF

This minimizes data transfer between MySQL and Dremio. Only the results of pushed-down operations cross the network.

Data Type Mapping

Key MySQL-to-Dremio type conversions:

MySQL	Dremio	Notes
INT / INTEGER	INTEGER
BIGINT	BIGINT	UNSIGNED converts to BIGINT
FLOAT	FLOAT
DOUBLE / REAL	DOUBLE
DECIMAL	DECIMAL
VARCHAR / TEXT / CHAR	VARCHAR	ENUM and SET also map to VARCHAR
DATE	DATE
DATETIME / TIMESTAMP	TIMESTAMP
TIME	TIME
BLOB / BINARY / VARBINARY	VARBINARY
BIT	BOOLEAN
TINYINT / SMALLINT / MEDIUMINT	INTEGER
YEAR	INTEGER

MySQL-specific types like JSON or GEOMETRY are not supported through the connector.

MySQL vs. Iceberg: When to Migrate

Keep data in MySQL when it's actively written and read by your application. Migrate historical or analytical datasets to Apache Iceberg tables in Dremio's Open Catalog when:

The data doesn't change often (closed orders, historical logs)
You need time travel (query the table as of any past timestamp)
You want automated performance management (compaction, manifest optimization)
You want Autonomous Reflections (Dremio auto-creates materializations based on query patterns)

For data that's still being written by your app, query it through the MySQL connector and create manual Reflections with a refresh schedule that matches your freshness needs.

AI-Powered Analytics on MySQL Data

Dremio AI Agent

The AI Agent lets business users ask questions about MySQL data in plain English. A marketing manager asks "How many high-value orders shipped last month?" and the Agent generates the correct SQL by reading your view's wiki descriptions. It understands "high-value" means total_amount > 500 and "shipped" means status = 'shipped' because you defined those in the semantic layer.

Dremio MCP Server

The Dremio MCP Server connects external AI chat clients (Claude, ChatGPT) to your MySQL data through Dremio with OAuth authentication:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs (e.g., https://claude.ai/api/mcp/auth_callback for Claude)
Connect using mcp.dremio.cloud/mcp/{project_id}

An e-commerce manager asks Claude "What's our average order value by region this quarter from MySQL?" and gets governed, accurate results — no SQL required.

AI SQL Functions

Use AI directly in queries against MySQL data:

-- Classify orders by likely customer intent
SELECT
  order_id,
  total_amount,
  order_tier,
  AI_CLASSIFY(
    'Based on this order, classify the likely purchase motivation',
    'Amount: $' || CAST(total_amount AS VARCHAR) || ', Status: ' || order_status || ', Tier: ' || order_tier,
    ARRAY['Impulse Buy', 'Planned Purchase', 'Bulk Order', 'Reorder']
  ) AS purchase_motivation
FROM analytics.gold.order_summary
WHERE order_status = 'completed';

-- Generate order analysis summaries
SELECT
  DATE_TRUNC('week', order_timestamp) AS week,
  COUNT(*) AS orders,
  SUM(total_amount) AS revenue,
  AI_GENERATE(
    'Write a one-sentence weekly sales summary',
    'Orders: ' || CAST(COUNT(*) AS VARCHAR) || ', Revenue: $' || CAST(SUM(total_amount) AS VARCHAR) || ', High Value Orders: ' || CAST(SUM(CASE WHEN order_tier = 'High Value' THEN 1 ELSE 0 END) AS VARCHAR)
  ) AS weekly_summary
FROM analytics.gold.order_summary
GROUP BY DATE_TRUNC('week', order_timestamp)
ORDER BY week DESC
LIMIT 12;

AI_CLASSIFY runs LLM inference inline, categorizing each order. AI_GENERATE produces narrative summaries. Both enrich MySQL data with AI in real time.

Reflections for Performance

MySQL is optimized for OLTP — row-level reads and writes. Analytical aggregation queries compete with application workloads. Dremio's Reflections offload these:

Navigate to the view in the Catalog
Click the Reflections tab
Choose Raw Reflection or Aggregation Reflection
Select columns and set the Refresh Interval
Click Save

BI tools get sub-second responses from Reflections. MySQL focuses on serving your application.

Governance on MySQL Data

MySQL has database-level grants but no column masking or row-level filtering. Dremio's Fine-Grained Access Control (FGAC) adds these:

Column masking: Mask customer email, payment details, or pricing from specific roles. A marketing analyst sees order counts but not individual customer data.
Row-level filtering: Restrict data by store, region, or department based on user role.
Unified policies: Same governance applies across MySQL, PostgreSQL, S3, and all other sources.

These policies apply across SQL Runner, BI tools (Arrow Flight/ODBC), AI Agent, and MCP Server.

Connect BI Tools via Arrow Flight

Arrow Flight provides 10-100x faster data transfer than JDBC/ODBC:

Tableau: Dremio connector for direct Arrow Flight access
Power BI: Dremio ODBC driver or native connector
Python/Pandas: pyarrow.flight for programmatic data access
Looker: Connect via JDBC
dbt: dbt-dremio for SQL-based transformations

All queries benefit from Reflections, governance, and the semantic layer.

VS Code Copilot Integration

Dremio's VS Code extension with Copilot integration lets developers query MySQL data from their IDE. Ask Copilot "Show me high-value orders from MySQL this week" and get SQL generated from your semantic layer.

When to Keep Data in MySQL vs. Migrate

Keep in MySQL: Transactional data for active applications, data with application-level foreign key constraints, operational data where real-time writes matter.

Migrate to Iceberg: Historical order archives, reporting data, data consumed by non-application tools, datasets where MySQL replication lag creates analytics latency. Migrated Iceberg tables get automatic compaction, time travel, and Autonomous Reflections.

For data staying in MySQL, create manual Reflections to offload analytical queries. For migrated Iceberg data, Dremio handles optimization automatically.

Get Started

MySQL users don't need to build ETL pipelines or provision a data warehouse to get analytical value from their data. Dremio Cloud connects to MySQL in minutes and gives you federation, acceleration, governance, and AI analytics on top.

Try Dremio Cloud free for 30 days and connect your MySQL databases.

Classify Your Data with SQL: A Hands-On Guide to Dremio's AI_CLASSIFY Function

Alex Merced — Sun, 01 Mar 2026 09:00:00 GMT

Most classification workflows require exporting data to Python, running a model, and importing results back into your warehouse. Dremio's AI_CLASSIFY function eliminates that entire pipeline. You write a SELECT statement, pass in your text and your categories, and the LLM assigns a label. The classified data stays in your lakehouse, governed and queryable immediately.

This tutorial walks you through a complete classification pipeline using a fresh Dremio Cloud account. You'll create sample customer feedback data, build a medallion architecture (Bronze → Silver → Gold), and use AI_CLASSIFY to categorize reviews by sentiment, support tickets by department, and product issues by urgency, all inside SQL.

What You'll Build

By the end of this tutorial, you'll have:

A customer feedback dataset with 50+ product reviews and 50+ support tickets
Bronze views that standardize raw data
Silver views that join reviews with ticket information
Gold views that use AI_CLASSIFY to add sentiment labels, department routing, and urgency tiers
An Iceberg table that persists your classified data for dashboards
Wiki metadata that enables the AI Agent to answer natural language questions about your classified data

Prerequisites

Dremio Cloud account — sign up free for 30 days with $400 in compute credits
AI enabled — go to Admin → Project Settings → Preferences → AI section and enable AI features
Model Provider configured — Dremio provides a hosted LLM by default, or you can connect your own (OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI) under the AI preferences

Note: Tables in the built-in Open Catalog use folder.subfolder.table_name without a catalog prefix. External sources use source_name.schema.table_name.

Understanding AI_CLASSIFY

AI_CLASSIFY sends text to a configured LLM and asks it to pick the best matching label from an array you provide. The function signature:

AI_CLASSIFY(
  [model_name VARCHAR,]
  prompt VARCHAR,
  categories ARRAY<VARCHAR|INT|FLOAT|BOOLEAN>
) → VARCHAR|INT|FLOAT|BOOLEAN

Parameters:

model_name (optional) — specify a particular model like 'gpt.4o'. Format is modelProvider.modelName. If omitted, Dremio uses your default configured model.
prompt — the text you want classified. This is typically a column value or a concatenation of columns that gives the LLM enough context.
categories — an ARRAY of possible labels. The LLM must return one of these values. Supports VARCHAR, INT, FLOAT, and BOOLEAN types.

The return type matches the array element type. If you pass ARRAY['Positive', 'Negative', 'Neutral'], you get a VARCHAR back. If you pass ARRAY[1, 2, 3, 4, 5], you get an INT.

Step 1: Create Your Folder Structure

Open the SQL Runner from the left sidebar in Dremio Cloud and run:

CREATE FOLDER IF NOT EXISTS aiclassifyexp;
CREATE FOLDER IF NOT EXISTS aiclassifyexp.feedback_data;
CREATE FOLDER IF NOT EXISTS aiclassifyexp.bronze;
CREATE FOLDER IF NOT EXISTS aiclassifyexp.silver;
CREATE FOLDER IF NOT EXISTS aiclassifyexp.gold;

This creates a namespace that simulates a customer feedback analytics pipeline with separate layers for raw data, standardized views, business logic, and final outputs.

Step 2: Seed Your Sample Data

Customer Reviews Table

This table simulates product reviews collected from an e-commerce platform. Each review includes the customer name, product, a star rating, and the actual review text that we'll classify.

CREATE TABLE aiclassifyexp.feedback_data.customer_reviews (
  review_id INT,
  customer_name VARCHAR,
  product_name VARCHAR,
  star_rating INT,
  review_text VARCHAR,
  review_date DATE
);

INSERT INTO aiclassifyexp.feedback_data.customer_reviews VALUES
(1, 'Sarah Chen', 'CloudSync Pro', 5, 'Absolutely love this product. Setup took 5 minutes and sync speeds are incredible. Best purchase this year.', '2025-08-15'),
(2, 'James Rodriguez', 'CloudSync Pro', 1, 'Terrible experience. Lost three days of data after the last update. Support was unhelpful and dismissive.', '2025-08-22'),
(3, 'Emily Watson', 'DataVault Enterprise', 4, 'Solid encryption and good performance. The UI could use some polish but the core functionality is reliable.', '2025-09-01'),
(4, 'Michael Brown', 'CloudSync Pro', 3, 'It works fine most of the time but crashes occasionally when syncing large folders. Average product.', '2025-09-05'),
(5, 'Lisa Park', 'DataVault Enterprise', 5, 'Our security team approved this after a thorough review. Encryption standards exceed our compliance requirements.', '2025-09-10'),
(6, 'David Kim', 'QuickReport', 2, 'The reports look nice but generation takes forever. For the price point there are faster alternatives.', '2025-09-12'),
(7, 'Anna Kowalski', 'QuickReport', 4, 'Great templates and easy export options. Scheduling could be more flexible but overall a good tool.', '2025-09-18'),
(8, 'Robert Taylor', 'CloudSync Pro', 1, 'Second time this month it corrupted my files during sync. Considering switching to a competitor.', '2025-09-20'),
(9, 'Maria Garcia', 'DataVault Enterprise', 5, 'Migrated 50TB without a single issue. The deduplication feature alone saved us $2000/month in storage.', '2025-09-25'),
(10, 'Tom Williams', 'QuickReport', 3, 'Decent for basic reports. Falls short on complex multi-source dashboards. Not bad, not great.', '2025-10-01'),
(11, 'Jennifer Lee', 'CloudSync Pro', 4, 'Fast reliable syncing across all our devices. The mobile app needs improvement though.', '2025-10-05'),
(12, 'Chris Martinez', 'DataVault Enterprise', 2, 'Way too complicated for a small team. We spent two weeks just on initial configuration.', '2025-10-08'),
(13, 'Rachel Adams', 'QuickReport', 5, 'Finally a reporting tool that non-technical people can use. Our marketing team builds their own reports now.', '2025-10-12'),
(14, 'Kevin Thompson', 'CloudSync Pro', 1, 'Billing issue: charged twice and it took three weeks to get a refund. Product aside the billing system is broken.', '2025-10-15'),
(15, 'Sophia Nguyen', 'DataVault Enterprise', 4, 'Strong security features and audit logging. Integration with our SSO provider was straightforward.', '2025-10-20'),
(16, 'Daniel Wilson', 'QuickReport', 3, 'Good for monthly summaries but real-time dashboards lag noticeably. Suitable for batch reporting only.', '2025-10-22'),
(17, 'Amanda Clark', 'CloudSync Pro', 5, 'Our entire team switched from Dropbox. The conflict resolution on shared files is leagues better.', '2025-10-25'),
(18, 'Brian Harris', 'DataVault Enterprise', 1, 'Critical vulnerability found in version 3.2. Support acknowledged it but the patch took 6 weeks.', '2025-10-28'),
(19, 'Michelle Lopez', 'QuickReport', 4, 'Clean interface and the PDF export quality is excellent. API access for automation would be a welcome addition.', '2025-11-01'),
(20, 'Steven Moore', 'CloudSync Pro', 2, 'Sync works but the desktop app uses 800MB of RAM just sitting in the background. Needs optimization.', '2025-11-05'),
(21, 'Laura Jackson', 'DataVault Enterprise', 5, 'Passed our SOC 2 audit partly because of DataVault detailed access logs. Worth every penny.', '2025-11-08'),
(22, 'Andrew White', 'QuickReport', 2, 'Crashed twice during a client presentation. Embarrassing and unacceptable for a paid product.', '2025-11-10'),
(23, 'Catherine Hall', 'CloudSync Pro', 4, 'Selective sync feature is a lifesaver for laptops with small drives. Smart storage management.', '2025-11-15'),
(24, 'Mark Allen', 'DataVault Enterprise', 3, 'Good product hampered by poor documentation. We figured out most features through trial and error.', '2025-11-18'),
(25, 'Jessica Young', 'QuickReport', 5, 'The scheduled email reports feature saved our ops team 10 hours per week. Simple and effective.', '2025-11-20');

Support Tickets Table

This table simulates a customer support system. Each ticket has a description written by the customer, a status, and a priority that was manually assigned. We'll use AI_CLASSIFY to automatically route these tickets by department.

CREATE TABLE aiclassifyexp.feedback_data.support_tickets (
  ticket_id INT,
  customer_name VARCHAR,
  product_name VARCHAR,
  ticket_description VARCHAR,
  manual_priority VARCHAR,
  ticket_status VARCHAR,
  created_date DATE,
  resolved_date DATE
);

INSERT INTO aiclassifyexp.feedback_data.support_tickets VALUES
(1001, 'James Rodriguez', 'CloudSync Pro', 'Lost all synced files after update 4.2.1. Need immediate recovery assistance.', 'Critical', 'Resolved', '2025-08-20', '2025-08-25'),
(1002, 'Kevin Thompson', 'CloudSync Pro', 'Charged $49.99 twice on my credit card for October subscription. Need refund for duplicate charge.', 'Medium', 'Resolved', '2025-10-14', '2025-11-04'),
(1003, 'Robert Taylor', 'CloudSync Pro', 'Files corrupted during sync for the second time. Happening with files over 500MB.', 'High', 'Open', '2025-09-19', NULL),
(1004, 'Chris Martinez', 'DataVault Enterprise', 'Cannot figure out how to configure SSO integration. Documentation references outdated menu options.', 'Medium', 'Resolved', '2025-10-07', '2025-10-10'),
(1005, 'Brian Harris', 'DataVault Enterprise', 'Security scan flagged CVE-2025-1234 in version 3.2 encryption module. When will this be patched?', 'Critical', 'Resolved', '2025-10-27', '2025-12-08'),
(1006, 'Andrew White', 'QuickReport', 'App crashes when rendering charts with more than 10000 data points. Happens consistently in Chrome.', 'High', 'Open', '2025-11-09', NULL),
(1007, 'Sarah Chen', 'CloudSync Pro', 'Would love to see a Linux desktop client. Currently only Windows and Mac are supported.', 'Low', 'Open', '2025-08-30', NULL),
(1008, 'David Kim', 'QuickReport', 'Report generation takes 45+ seconds for simple 3-page reports. Was faster in the previous version.', 'Medium', 'Open', '2025-09-13', NULL),
(1009, 'Emily Watson', 'DataVault Enterprise', 'Need to add 50 new users to our plan. What are the volume discount options?', 'Low', 'Resolved', '2025-09-03', '2025-09-05'),
(1010, 'Steven Moore', 'CloudSync Pro', 'Desktop app consuming excessive memory (800MB+). Running Windows 11 with 16GB RAM.', 'Medium', 'Open', '2025-11-04', NULL),
(1011, 'Lisa Park', 'DataVault Enterprise', 'Can we get a custom retention policy for healthcare compliance? HIPAA requires 7-year retention.', 'Medium', 'Resolved', '2025-09-12', '2025-09-20'),
(1012, 'Tom Williams', 'QuickReport', 'How do I connect QuickReport to a PostgreSQL database? Only seeing MySQL option in connectors.', 'Low', 'Resolved', '2025-10-02', '2025-10-03'),
(1013, 'Mark Allen', 'DataVault Enterprise', 'API documentation has broken links on the authentication section. Pages return 404.', 'Low', 'Open', '2025-11-17', NULL),
(1014, 'Michael Brown', 'CloudSync Pro', 'Selective sync keeps re-enabling folders I excluded. Happens after every app restart.', 'Medium', 'Open', '2025-09-06', NULL),
(1015, 'Daniel Wilson', 'QuickReport', 'Real-time dashboard shows data that is 15 minutes stale. Expected near real-time refresh.', 'High', 'Open', '2025-10-23', NULL),
(1016, 'Anna Kowalski', 'QuickReport', 'Can you add a dark mode option? The white background is hard on the eyes during evening work.', 'Low', 'Open', '2025-09-19', NULL),
(1017, 'Sophia Nguyen', 'DataVault Enterprise', 'Our SSO integration broke after your last update. 200 users locked out for 4 hours.', 'Critical', 'Resolved', '2025-10-21', '2025-10-21'),
(1018, 'Jennifer Lee', 'CloudSync Pro', 'Mobile app on iOS frequently logs me out. Have to re-authenticate 3-4 times per day.', 'Medium', 'Open', '2025-10-06', NULL),
(1019, 'Rachel Adams', 'QuickReport', 'Love the product! Any plans for a Slack integration to send report summaries to channels?', 'Low', 'Open', '2025-10-13', NULL),
(1020, 'Amanda Clark', 'CloudSync Pro', 'Conflict resolution dialog is confusing. Hard to tell which version is newer when filenames match.', 'Medium', 'Resolved', '2025-10-26', '2025-10-30'),
(1021, 'Catherine Hall', 'CloudSync Pro', 'Bandwidth throttling feature needed. Sync saturates our office internet during business hours.', 'Medium', 'Open', '2025-11-16', NULL),
(1022, 'Maria Garcia', 'DataVault Enterprise', 'Deduplication incorrectly merged two different client folders. Data was mixed across accounts.', 'Critical', 'Resolved', '2025-09-26', '2025-09-28'),
(1023, 'Laura Jackson', 'DataVault Enterprise', 'Need export of all access logs for the past 12 months for our annual SOC 2 audit.', 'Medium', 'Resolved', '2025-11-09', '2025-11-11'),
(1024, 'Jessica Young', 'QuickReport', 'Scheduled reports occasionally skip a week. No error notification when this happens.', 'High', 'Open', '2025-11-21', NULL),
(1025, 'Michelle Lopez', 'QuickReport', 'Please add an API endpoint for programmatic report generation. We want to automate monthly client reports.', 'Low', 'Open', '2025-11-02', NULL);

Step 3: Build Bronze Views

Bronze views standardize column names and data types without applying business logic. This creates a consistent foundation for downstream analysis.

The reviews table needs its DATE column cast to TIMESTAMP for consistent joins later. The tickets table also needs date casting, and we rename manual_priority to assigned_priority to distinguish it from AI-generated classifications.

CREATE OR REPLACE VIEW aiclassifyexp.bronze.v_reviews AS
SELECT
  review_id,
  customer_name,
  product_name,
  star_rating,
  review_text,
  CAST(review_date AS TIMESTAMP) AS review_timestamp
FROM aiclassifyexp.feedback_data.customer_reviews;

CREATE OR REPLACE VIEW aiclassifyexp.bronze.v_tickets AS
SELECT
  ticket_id,
  customer_name,
  product_name,
  ticket_description,
  manual_priority AS assigned_priority,
  ticket_status,
  CAST(created_date AS TIMESTAMP) AS created_timestamp,
  CAST(resolved_date AS TIMESTAMP) AS resolved_timestamp
FROM aiclassifyexp.feedback_data.support_tickets;

Step 4: Build Silver Views

This Silver view joins reviews with related support tickets for the same customer and product. This gives us a combined picture: what did the customer say in their review, and did they also file a support ticket? The LEFT JOIN ensures we keep all reviews even if the customer never opened a ticket.

CREATE OR REPLACE VIEW aiclassifyexp.silver.v_customer_feedback AS
SELECT
  r.review_id,
  r.customer_name,
  r.product_name,
  r.star_rating,
  r.review_text,
  r.review_timestamp,
  t.ticket_id,
  t.ticket_description,
  t.assigned_priority,
  t.ticket_status,
  t.created_timestamp AS ticket_created,
  t.resolved_timestamp AS ticket_resolved,
  CASE WHEN t.ticket_id IS NOT NULL THEN 'Yes' ELSE 'No' END AS has_support_ticket
FROM aiclassifyexp.bronze.v_reviews r
LEFT JOIN aiclassifyexp.bronze.v_tickets t
  ON r.customer_name = t.customer_name
  AND r.product_name = t.product_name;

Step 5: Build Gold Views with AI_CLASSIFY

This is where the AI functions do real work. Each Gold view applies AI_CLASSIFY to categorize text that would otherwise require manual review or an external ML pipeline.

Gold View 1: Sentiment Classification

This view classifies every review as Positive, Negative, or Neutral. Instead of relying solely on star ratings (which can be inconsistent with the actual text), the LLM reads the full review and assigns a sentiment label. We concatenate the product name with the review text to give the model full context.

CREATE OR REPLACE VIEW aiclassifyexp.gold.v_review_sentiment AS
SELECT
  review_id,
  customer_name,
  product_name,
  star_rating,
  review_text,
  review_timestamp,
  AI_CLASSIFY(
    'Classify the sentiment of this product review: ' || review_text,
    ARRAY['Positive', 'Negative', 'Neutral']
  ) AS ai_sentiment,
  has_support_ticket
FROM aiclassifyexp.silver.v_customer_feedback;

Notice that we keep both star_rating and ai_sentiment. This lets you compare the two signals. A 3-star review with "Negative" AI sentiment suggests the customer is more frustrated than the rating alone indicates.

Gold View 2: Ticket Department Routing

This view uses AI_CLASSIFY to automatically route support tickets to the right department based on the ticket description. Instead of a human reading every ticket and assigning it, the LLM reads the description and selects from four departments.

CREATE OR REPLACE VIEW aiclassifyexp.gold.v_ticket_routing AS
SELECT
  ticket_id,
  customer_name,
  product_name,
  ticket_description,
  assigned_priority,
  ticket_status,
  created_timestamp,
  resolved_timestamp,
  AI_CLASSIFY(
    'Based on this support ticket, which department should handle it: ' || ticket_description,
    ARRAY['Billing', 'Technical Support', 'Feature Request', 'Account Management']
  ) AS ai_department,
  AI_CLASSIFY(
    'Rate the urgency of this support ticket: ' || ticket_description,
    ARRAY['Critical', 'High', 'Medium', 'Low']
  ) AS ai_urgency
FROM aiclassifyexp.bronze.v_tickets;

This view applies two separate AI_CLASSIFY calls on each row: one for department routing and one for urgency. You can compare ai_urgency against the manually assigned assigned_priority to find tickets where human triage may have underestimated or overestimated severity.

Using Numeric Categories

AI_CLASSIFY also supports numeric arrays. If you want a 1-5 satisfaction score instead of text labels:

SELECT
  review_id,
  review_text,
  AI_CLASSIFY(
    'Rate customer satisfaction from 1 (very dissatisfied) to 5 (very satisfied): ' || review_text,
    ARRAY[1, 2, 3, 4, 5]
  ) AS ai_satisfaction_score
FROM aiclassifyexp.bronze.v_reviews;

The LLM returns an INT because the array contains integers. This is useful when you need numeric scores for aggregation, averages, or trend analysis.

Persisting Results with CTAS

AI function calls consume LLM tokens on every query execution. For dashboards or reports that run the same classification repeatedly, materialize the results into an Iceberg table with CREATE TABLE AS SELECT (CTAS):

CREATE TABLE aiclassifyexp.gold.classified_reviews AS
SELECT * FROM aiclassifyexp.gold.v_review_sentiment;

This creates a physical Iceberg table with the AI classifications baked in. Subsequent queries against classified_reviews are standard SQL queries with no LLM cost. Refresh the table periodically (daily, weekly) as new reviews come in by running CTAS again with CREATE OR REPLACE TABLE.

Managing AI Workloads

AI function queries are more resource-intensive than standard SQL. Dremio provides engine routing to isolate these workloads:

-- Dremio provides these routing functions for workload management:
-- query_calls_ai_functions() — returns true if the query uses AI functions
-- query_has_attribute('AI_FUNCTIONS') — same check, different syntax

In your Dremio Cloud project settings, you can create engine routing rules that automatically direct queries containing AI functions to a dedicated engine. This prevents a large classification batch job from competing with your executive dashboards for compute resources. Set up a separate engine with appropriate scaling for AI workloads, and create a routing rule using query_calls_ai_functions() to send AI queries there automatically.

Choosing Your Model Provider

The optional model_name parameter lets you target specific models for different tasks:

-- Use a specific model for classification
SELECT AI_CLASSIFY(
  'openai.gpt-4o',
  'Classify this ticket: ' || ticket_description,
  ARRAY['Billing', 'Technical Support', 'Feature Request']
) AS department
FROM aiclassifyexp.bronze.v_tickets;

Dremio supports multiple providers: OpenAI, Anthropic, Google Gemini, AWS Bedrock, and Azure OpenAI. You configure providers in Admin → Project Settings → Preferences → AI. The format is providerName.modelName, where providerName is the name you gave the provider during setup.

If you skip model_name, Dremio uses your default model. For most classification tasks, the default model works well. Specifying a model makes sense when you need a particular model's strengths (like a smaller, faster model for simple sentiment vs. a larger model for nuanced multi-class categorization).

Step 6: Enable AI-Generated Wikis and Tags

Good metadata makes the AI Agent more accurate when answering natural language questions. Here's how to add context to your Gold views:

Click Admin in the left sidebar, then go to Project Settings.
Select the Preferences tab.
Scroll to the AI section and enable Generate Wikis and Labels.
Go to the Catalog and navigate to your Gold views under aiclassifyexp.gold.
Click the Edit button (pencil icon) next to the desired view.
In the Details tab, find the Wiki section and click Generate Wiki. Do the same for the Tags section by clicking Generate Tags.
Repeat for each Gold view.

To enhance the generated wiki with additional business context, copy the output into the AI Agent on the homepage and ask it to produce an improved version in a markdown code block. For example, ask the Agent to add details like "Positive sentiment reviews are candidates for testimonial collection. Negative sentiment reviews with support tickets should trigger a customer success outreach." Copy the Agent's refined output and paste it back into the wiki editor.

Wikis and labels are the context that Dremio's AI Agent reads before generating SQL. Better metadata produces more accurate natural language responses.

Step 7: Ask Questions with the AI Agent

With your classified data and enriched wikis in place, navigate to the AI Agent on the Dremio homepage and try these prompts:

"Which products have the most negative reviews?"

The Agent queries v_review_sentiment, filters by ai_sentiment = 'Negative', groups by product_name, and returns a count. You'll see which products need attention based on LLM-analyzed sentiment rather than just star ratings.

"Show me a chart of ticket routing by department"

The Agent queries v_ticket_routing, groups by ai_department, and generates a bar chart showing how tickets distribute across Billing, Technical Support, Feature Request, and Account Management.

"List all critical urgency tickets that are still open, ordered by creation date"

The Agent filters v_ticket_routing for ai_urgency = 'Critical' and ticket_status = 'Open', sorts by created_timestamp, and returns the results. This surfaces tickets that AI flagged as critical but haven't been resolved.

"Create a chart showing sentiment distribution by product and whether the customer has a support ticket"

The Agent creates a multi-dimensional visualization from v_review_sentiment, cross-referencing product_name, ai_sentiment, and has_support_ticket. This reveals patterns like "CloudSync Pro has the most negative reviews among customers who also filed support tickets."

Why Apache Iceberg Matters

All the tables you created in this tutorial are Apache Iceberg tables stored in Dremio's built-in Open Catalog. Iceberg provides ACID transactions, schema evolution, and time travel, but the performance benefits are especially relevant for AI-classified data.

Iceberg vs. Federated for AI Workloads

Keep data federated when: Your classification needs real-time source data; for example, classifying support tickets as they arrive from a live PostgreSQL database. Use manual Reflections with a short refresh interval to accelerate federated queries.

Migrate to Iceberg when: You're running batch classification jobs on historical data. The CTAS approach above creates Iceberg tables. Iceberg's automated performance management (compaction, manifest optimization, clustering) keeps these growing tables fast. Autonomous Reflections can auto-create pre-computed materializations based on how your dashboards query the classified data.

Cost Optimization Pattern

Run AI_CLASSIFY once via CTAS to materialize results. Build Reflections on the materialized table for dashboard queries. This pattern means you pay for LLM tokens once during classification, and all subsequent analytical queries hit cached Reflections at zero LLM cost.

Next Steps

Connect real data sources — replace the feedback_data folder with federated connections to your actual CRM, support platform, and review system
Add Fine-Grained Access Control (FGAC) — mask customer names or PII in classified results so analysts see sentiment patterns without accessing personal data
Experiment with Boolean classification — use ARRAY[true, false] for binary decisions like "Is this review about a security concern?" or "Does this ticket mention data loss?"
Scale with Reflections — create Reflections on your materialized classification tables to accelerate dashboard queries

If you're running manual classification processes today, whether it's tagging support tickets, scoring reviews, or categorizing feedback, AI_CLASSIFY replaces those workflows with a single SQL query. The classification runs inside the same platform where your data lives, governed by the same access controls, and immediately available to every BI tool and AI agent connected to Dremio.

Try Dremio Cloud free for 30 days and start classifying your data with SQL.

Connect PostgreSQL to Dremio Cloud: Query, Federate, and Accelerate Your Data

Alex Merced — Sun, 01 Mar 2026 09:00:00 GMT

PostgreSQL powers more production applications than almost any other open-source database. It's where your customer records, transaction logs, product catalogs, and operational data live. But running analytics directly against PostgreSQL creates problems: heavy analytical queries compete with transactional workloads, cross-database joins require custom ETL, and your data team can't access PostgreSQL data alongside data in S3, Snowflake, or other systems without building pipelines.

Dremio Cloud solves this by connecting directly to PostgreSQL and querying it in place. No data movement, no ETL pipelines, no replica databases. You write SQL in Dremio, and it pushes filtering and aggregation work back to PostgreSQL when possible, fetches only the results, and lets you join that data with any other connected source in the same query.

This guide walks through connecting PostgreSQL to Dremio Cloud, from prerequisites to your first federated query.

Why PostgreSQL Users Need Dremio

PostgreSQL is an excellent transactional database, but it wasn't designed for the analytics patterns that modern teams need. Here are the problems Dremio solves:

Cross-source analytics without pipelines. Your customer data is in PostgreSQL, your clickstream data is in S3, and your revenue data is in Snowflake. Without Dremio, joining these datasets requires building ETL pipelines to centralize everything into one system. With Dremio, you connect all three as sources and write a single SQL query that joins across them. Dremio handles the federation.

Protect production performance. Running heavy GROUP BY queries or full-table scans against your production PostgreSQL instance can degrade application performance. Dremio's Reflections solve this by creating pre-computed materializations of your most common analytical queries. After the first query, subsequent queries hit the Reflection instead of PostgreSQL, eliminating load on your production database.

Business context for AI. Raw PostgreSQL tables have technical column names like cust_id and txn_amt. Dremio's semantic layer lets you create views that rename and restructure these columns with business logic, then attach wiki descriptions and labels. When your team asks Dremio's AI Agent "Who are our highest-value customers?", the Agent understands what "highest-value" means because you've defined it in the semantic layer.

Governance without modifying PostgreSQL. Dremio's Fine-Grained Access Control (FGAC) lets you mask sensitive columns (Social Security numbers, email addresses) and filter rows based on user roles. You don't need to modify PostgreSQL permissions or create restricted database views — the governance layer lives in Dremio and applies across all tools and users.

What You Need Before Connecting

Before configuring the connection in Dremio, make sure you have:

PostgreSQL hostname or IP address — the network address of your database server
Port number — PostgreSQL defaults to 5432
Database name — the specific database you want to connect
Username and password — credentials for a user with read access to the tables you want to query
Network accessibility — Dremio Cloud connects to your PostgreSQL instance over the public internet by default. Ensure port 5432 (or your custom port) is open in your AWS Security Group, Azure NSG, or firewall rules

If your PostgreSQL instance is in a private subnet (common for production databases), you'll need to configure networking to allow Dremio Cloud to reach it. Check Dremio's network connectivity documentation for options.

Dremio Cloud account: Sign up at dremio.com/get-started for a free 30-day trial with $400 in compute credits.

Step-by-Step: Connect PostgreSQL to Dremio Cloud

1. Add a New Source

In the Dremio console, click the "+" button in the left sidebar and select PostgreSQL from the database source types. Alternatively, navigate to Databases in the data panel and click Add database.

2. Configure General Settings

Fill in the connection details:

Name: Enter a descriptive name for this source (e.g., production-postgres or crm-database). This name will appear in your SQL queries when referencing tables from this source. Note: the name cannot include /, :, [, or ].
Host: Enter your PostgreSQL hostname (e.g., my-db.cluster-abc123.us-east-1.rds.amazonaws.com).
Port: Enter the port number. The default is 5432.
Database: Enter the database name you want to connect to.
Encrypt connection: Toggle this on to use SSL encryption between Dremio and PostgreSQL. Recommended for production connections, especially when connecting over the internet.

3. Set Authentication

Choose one of two authentication methods:

Master Authentication (default): Provide a username and password directly. This is the simplest option — enter the credentials for a PostgreSQL user that has SELECT permissions on the tables you want to query.

Secret Resource URL: Instead of storing the password in Dremio, provide an AWS Secrets Manager ARN (e.g., arn:aws:secretsmanager:us-west-2:123456789012:secret:my-rds-secret-VNenFy). Dremio fetches the password from Secrets Manager at connection time. This is the preferred option for production deployments because it centralizes credential management and supports rotation.

4. Configure Advanced Options (Optional)

The advanced options let you fine-tune connection behavior:

Setting	What It Does	Default
Record fetch size	Number of rows Dremio fetches per batch. Set to 0 for automatic.	200
Maximum Idle Connections	How many idle connections Dremio maintains to PostgreSQL.	8
Connection Idle Time	Seconds before an idle connection is closed.	60
Encryption Validation Mode	When SSL is enabled: validate certificate + hostname, certificate only, or no validation.	Validate both
Connection Properties	Custom key-value pairs for JDBC connection parameters.	None

For most users, the defaults work fine. If you're connecting to an Amazon RDS or Aurora instance, the default SSL settings are compatible.

5. Set Reflection Refresh Schedule

This controls how often Dremio refreshes pre-computed Reflections built on PostgreSQL data:

Refresh every: How often Reflections update (hours, days, or weeks). More frequent refreshes mean fresher data but more queries against PostgreSQL.
Expire after: How long before unused Reflections are automatically removed.

For operational PostgreSQL data that changes throughout the day, a refresh interval of 1-4 hours is typical. For historical data that rarely changes, daily or weekly is sufficient.

6. Configure Metadata Refresh

These settings control how often Dremio checks PostgreSQL for new or changed tables:

Dataset Discovery (Fetch every): How often Dremio looks for new tables or schema changes. Default is 1 hour.
Dataset Details (Fetch every): How often Dremio refreshes detailed metadata for tables you've already queried. Default is 1 hour.

7. Set Privileges and Save

Optionally, restrict which Dremio users or roles can access this PostgreSQL source. Click Save to create the connection.

Query PostgreSQL Data from Dremio

Once connected, your PostgreSQL database appears as a source in Dremio's SQL Runner. Browse the source to see schemas and tables, then query them directly:

SELECT customer_id, first_name, last_name, signup_date
FROM "production-postgres".public.customers
WHERE signup_date > '2024-01-01'
ORDER BY signup_date DESC
LIMIT 100;

The source name (production-postgres) is the name you gave the source. PostgreSQL schemas appear as sub-folders, and tables appear within those schemas.

Federate PostgreSQL with Other Sources

The real value appears when you combine PostgreSQL data with other sources. Here's an example that joins PostgreSQL customer data with S3 clickstream data:

SELECT
  c.customer_id,
  c.first_name || ' ' || c.last_name AS customer_name,
  c.segment,
  COUNT(e.event_id) AS total_events,
  SUM(CASE WHEN e.event_type = 'purchase' THEN 1 ELSE 0 END) AS purchases
FROM "production-postgres".public.customers c
LEFT JOIN "s3-clickstream".events.user_events e
  ON c.customer_id = e.user_id
GROUP BY c.customer_id, c.first_name, c.last_name, c.segment
ORDER BY purchases DESC;

Dremio pushes the filter and projection operations to PostgreSQL (this is called predicate pushdown), fetches only the matching rows, then joins them with the S3 data in Dremio's query engine. PostgreSQL handles what it's good at (filtering indexed columns), and Dremio handles the cross-source join.

Build a Semantic Layer Over PostgreSQL

Create views to give your PostgreSQL data business-friendly names and logic:

CREATE VIEW analytics.gold.customer_overview AS
SELECT
  c.customer_id,
  c.first_name || ' ' || c.last_name AS full_name,
  c.email,
  c.segment AS customer_segment,
  c.signup_date,
  CASE
    WHEN c.segment = 'Enterprise' AND c.lifetime_value > 50000 THEN 'Strategic'
    WHEN c.lifetime_value > 10000 THEN 'High Value'
    ELSE 'Standard'
  END AS account_tier
FROM "production-postgres".public.customers c;

Then attach wiki descriptions and labels through the Catalog (edit pencil icon → Details tab → Generate Wiki/Tags) so the AI Agent understands the data when users ask natural language questions.

Predicate Pushdown: What Dremio Offloads to PostgreSQL

Dremio doesn't download entire PostgreSQL tables and process them locally. When possible, it pushes operations back to PostgreSQL to minimize data transfer. PostgreSQL supports an extensive set of pushdowns in Dremio, including:

Logical operators: AND, OR, NOT
Comparisons: =, !=, <, >, <=, >=, BETWEEN, IN, LIKE, IS NULL, IS NOT NULL
Aggregations: SUM, AVG, COUNT, MIN, MAX, STDDEV, MEDIAN, VAR_POP
Math functions: ABS, CEIL, FLOOR, ROUND, MOD, SQRT, POWER, LOG
String functions: CONCAT, SUBSTR, LENGTH, LOWER, UPPER, TRIM, REPLACE, REVERSE
Date functions: DATE_ADD, DATE_SUB, DATE_TRUNC (day, hour, month, quarter, year), EXTRACT

This means a query like SELECT department, AVG(salary) FROM postgres.hr.employees WHERE hire_date > '2023-01-01' GROUP BY department runs mostly on PostgreSQL — Dremio sends the filter, aggregation, and grouping to Postgres and only transfers the summarized result.

Accelerate PostgreSQL Queries with Reflections

For queries that run frequently, create Reflections to avoid hitting PostgreSQL repeatedly:

Build a view over your PostgreSQL data.
In the Catalog, select the view and create a Reflection.
Choose the columns and aggregations to include.
Set the refresh interval (how often Dremio re-queries PostgreSQL to update the Reflection).

After the Reflection is built, Dremio's query optimizer automatically routes matching queries to the Reflection instead of PostgreSQL. Your analysts see the same tables and write the same SQL — the acceleration is transparent.

This is particularly valuable for dashboard queries. BI tools like Tableau or Power BI connected to Dremio via Arrow Flight/ODBC get sub-second response times from Reflections, even though the source data lives in PostgreSQL.

Data Type Mapping

Dremio automatically maps PostgreSQL types to Dremio types. The key mappings to know:

PostgreSQL	Dremio	Notes
BIGINT / BIGSERIAL	BIGINT
INT / SERIAL	INTEGER
NUMERIC	DECIMAL	Preserves precision
VARCHAR / TEXT / CHAR	VARCHAR
BOOLEAN / BIT	BOOLEAN
DATE	DATE
TIMESTAMP / TIMESTAMPTZ	TIMESTAMP	Timezone-aware types convert
FLOAT4 / FLOAT8	FLOAT / DOUBLE
BYTEA	VARBINARY
MONEY	DOUBLE	Converted to numeric

Most types map directly. If you use PostgreSQL-specific types like JSONB, ARRAY, or HSTORE, those are not supported in Dremio's connector and won't appear in query results.

AI-Powered Analytics on PostgreSQL Data

Dremio AI Agent

The built-in AI Agent lets users ask questions about PostgreSQL data in plain English. Instead of writing SQL, a business user asks "Who are our highest-value enterprise customers?" and the Agent generates the correct query by reading the wiki descriptions attached to your semantic layer views. The Agent understands that "highest-value" maps to lifetime_value and "enterprise" maps to segment = 'Enterprise' because you've defined it in the view's wiki.

Dremio MCP Server

The Dremio MCP Server connects external AI chat clients — Claude, ChatGPT, and others — to your PostgreSQL data through Dremio. The hosted MCP Server provides OAuth authentication that propagates user identity and authorization for every interaction:

Create a Native OAuth application in Dremio Cloud
Configure redirect URLs for your AI client (e.g., https://claude.ai/api/mcp/auth_callback)
Connect using mcp.dremio.cloud/mcp/{project_id}

A sales director can ask Claude "Show me our strategic account customers who signed up in Q1" and get governed, accurate results from your PostgreSQL data without SQL.

AI SQL Functions

Use AI directly in queries against PostgreSQL data:

-- Classify customers based on their profile data
SELECT
  full_name,
  customer_segment,
  account_tier,
  AI_CLASSIFY(
    'Based on this customer profile, predict their likely next action',
    'Customer: ' || full_name || ', Segment: ' || customer_segment || ', Tier: ' || account_tier,
    ARRAY['Upsell Opportunity', 'Renewal Risk', 'Expansion Ready', 'Stable']
  ) AS predicted_action
FROM analytics.gold.customer_overview
WHERE account_tier IN ('Strategic', 'High Value');

-- Generate personalized engagement plans
SELECT
  full_name,
  AI_GENERATE(
    'Write a one-sentence personalized engagement recommendation',
    'Customer: ' || full_name || ', Segment: ' || customer_segment || ', Tier: ' || account_tier || ', Signup: ' || CAST(signup_date AS VARCHAR)
  ) AS engagement_recommendation
FROM analytics.gold.customer_overview
WHERE account_tier = 'Strategic';

AI_CLASSIFY categorizes data with LLM inference inside SQL. AI_GENERATE produces text. AI_SIMILARITY (not shown) finds semantic matches between text fields. All run directly in your query.

Get Started

If you run PostgreSQL for your application data and want to include it in cross-source analytics, AI-driven queries, or governed dashboards without building ETL pipelines, Dremio Cloud is the fastest path.

Try Dremio Cloud free for 30 days and connect your PostgreSQL instance in under 5 minutes.

Data Engineering Best Practices: The Complete Checklist

Alex Merced — Wed, 18 Feb 2026 18:00:00 GMT

Best practices documents are easy to write and hard to use. They list principles without context, advice without prioritization, and rules without explaining when to break them. This one is different. It's a practical, tool-agnostic checklist organized by the categories that matter most — with each item tied to a specific outcome.

Use this as a recurring audit. Run through it quarterly. Any unchecked item is either a technical debt item or a conscious tradeoff. Know which is which.

Pipeline Design

[ ] Separate ingestion from transformation. Raw data lands unchanged. Business logic runs separately. This lets you replay raw data and isolate failures.
[ ] Model pipelines as DAGs. Each stage has explicit inputs and outputs. Independent stages run in parallel. Failed stages retry alone.
[ ] Make dependencies explicit. If pipeline B needs the output of pipeline A, declare that dependency in your orchestrator. Don't rely on timing assumptions.
[ ] Use sensors or triggers for scheduling. Wait for data to arrive, not for the clock to hit a certain time. Data-driven triggers are more reliable than cron jobs.
[ ] Keep stages single-purpose. An ingestion stage should not also validate, transform, and load. Each stage does one thing and does it well.

Data Quality

[ ] Validate schema at ingestion. Compare incoming data against expected column names, types, and nullability. Catch schema drift before it propagates.
[ ] Check completeness. Required fields have no nulls. If customer_id is nullable in your orders table, downstream joins will silently lose rows.
[ ] Enforce uniqueness. Primary keys have no duplicates. Run dedup checks after every load. Double-counted records are worse than missing records.
[ ] Quarantine bad records. Route validation failures to a quarantine table with metadata (which check failed, when, the original record). Never drop records silently.
[ ] Track quality metrics. Null rates, duplicate rates, and range violations tracked per pipeline, per day. Trend these metrics to catch gradual degradation.

Reliability and Idempotency

[ ] Make every pipeline idempotent. Running the same job twice produces the same result. Use partition overwrite or MERGE — never blind INSERT.
[ ] Implement retry with backoff. Transient failures (network, API limits) resolve themselves. Retry 3-5 times with exponential backoff before alerting.
[ ] Use dead-letter queues. Records that can't be processed go to a queue for inspection, not to /dev/null.
[ ] Checkpoint progress. After processing each batch or partition, record what's done. On failure, resume from the last checkpoint.
[ ] Design for failure. Every component will fail. Define the expected behavior for each failure mode: retry, skip and log, alert, or halt.

Schema Management

[ ] Treat your schema as an API. Column names are fields. Tables are endpoints. Consumers are clients. Changing the schema without coordination is as bad as changing an API without versioning.
[ ] Use additive-only changes. Add new columns. Never remove or rename columns without a deprecation period.
[ ] Enforce contracts at boundaries. Validate that incoming schema matches expectations at ingestion. Validate that outgoing schema matches consumer contracts at serving.
[ ] Version breaking changes. When a schema must change incompatibly, version it (v1, v2). Let consumers migrate on their own schedule.
[ ] Document every column. Column name, type, description, source, owner. If an engineer can't find this information in under 30 seconds, it's not documented.

Testing and Validation

[ ] Run schema tests on every pipeline execution. Column existence, data types, not-null constraints. These are fast, cheap, and catch the most common problems.
[ ] Run uniqueness and null checks on primary keys. The two most impactful data quality tests. Add them today.
[ ] Compare row counts against baselines. Alert when today's count deviates by more than 20% from the trailing average. Catches missing data and unexpected volume spikes.
[ ] Test transformation logic with fixtures. Small, known-good input datasets with expected outputs. Run these in CI before deploying pipeline changes.
[ ] Add regression tests for key business metrics. Total revenue, distinct customer count, and other critical aggregations compared against previous runs.

Observability and Monitoring

[ ] Track data freshness per table. The timestamp of the most recent row. Alert when it exceeds the SLA. This single metric catches more problems than any other.
[ ] Alert on business impact, not every error. SLA violations, quality regressions, and anomalous volume changes are alerts. Transient retries and expected maintenance are not.
[ ] Use structured logging. JSON-formatted log entries with pipeline name, stage, batch ID, timestamp, row count, and status. Searchable, parseable, filterable.
[ ] Build data lineage. Know where each table's data comes from and where it goes. Column-level lineage turns "the numbers are wrong" from a half-day investigation into a 10-minute graph traversal.
[ ] Review observability quarterly. Are alerts still relevant? Are thresholds still accurate? Are dashboards still used? Trim unactionable alerts and update stale baselines.

What to Do Next

Print this checklist. Walk through it with your team in a 30-minute meeting. Check what's already in place, identify the three highest-impact unchecked items, and schedule them as engineering work — not aspirational goals on a wiki page. Best practices only matter when they're implemented.

Try Dremio Cloud free for 30 days

Data Modeling Best Practices: 7 Mistakes to Avoid

Alex Merced — Wed, 18 Feb 2026 18:00:00 GMT

A bad data model doesn't announce itself. It hides behind slow dashboards, conflicting numbers, confused analysts, and AI agents that generate wrong SQL. By the time someone identifies the model as the root cause, the team has already built dozens of reports on top of it.

Here are seven modeling mistakes that create downstream pain — and how to avoid each one.

Mistake 1: No Defined Grain

The grain declares what one row in a fact table represents. "One row per order line item." "One row per daily user session." "One row per monthly account balance."

Without a declared grain, aggregation produces wrong numbers. If some rows represent individual transactions and others represent daily summaries, a SUM query double-counts or under-counts depending on the mix.

Fix: Before designing any fact table, write down the grain in one sentence. Share it with your team. If you can't state the grain clearly, the table isn't ready for production.

Mistake 2: Cryptic Naming

Columns named c1, dt, amt, flg, and cat_cd save keystrokes during development but cost hours during analysis. Every analyst who encounters these names must either read the ETL code, ask the engineer, or guess.

AI agents have the same problem. An agent asked to calculate "total revenue" can't identify the right column if it's called amt3 instead of revenue_usd.

Fix: Use descriptive, business-friendly names. customer_name, order_date, revenue_usd, is_active, product_category. Include units where ambiguous (weight_kg, duration_minutes). Use snake_case consistently.

Mistake 3: Skipping the Conceptual Model

Going straight from a stakeholder request to CREATE TABLE skips the alignment step. Engineers build what they understand from the request. Stakeholders assumed something different. The gap surfaces weeks or months later when reports don't match expectations.

Fix: For every new business domain, create a conceptual model first. List the entities, name the relationships, and get business stakeholder sign-off before writing any SQL.

Mistake 4: Over-Normalizing for Analytics

Third Normal Form (3NF) is correct for transactional systems where writes are frequent and consistency matters. Applied to an analytics workload, it creates queries with 10-15 joins that run slowly and break easily.

Fix: Separate your transactional model from your analytical model. Keep the OLTP system in 3NF. Build a denormalized star schema (or a set of wide views) for analytics. Different workloads deserve different models.

Mistake 5: Under-Documenting

A data model without documentation is a puzzle that only its creator can solve. And even they forget the details after a few months.

Without documentation, every new team member reverse-engineers the model from scratch. Every AI agent generates SQL based on guesses. Every analyst interprets column meanings differently, leading to metric discrepancies that take weeks to reconcile.

Fix: Document at three levels:

Column level: What does each column mean? Where does it come from?
Table level: What grain does this table use? Who maintains it?
Model level: How do tables connect? What business process does this model represent?

Platforms like Dremio make this practical with built-in Wikis for every dataset and Labels for classification (PII, Certified, Raw, Deprecated). The documentation lives next to the data, not in a separate spreadsheet that goes stale.

Mistake 6: One Model for Every Workload

A model designed for a transactional application doesn't serve analytics well. A model designed for analytics doesn't serve a machine learning feature store well. Trying to make one model serve every use case leads to compromises that serve no use case well.

Fix: Build purpose-specific models layered on top of shared source data. The Medallion Architecture does this naturally:

Bronze: Raw data from sources (shared foundation)
Silver: Business logic layer (shared across analytics and ML)
Gold: Purpose-built views (one for dashboards, one for ML features, one for AI agents)

Each Gold view is tailored to its consumer without duplicating the transformation logic in Silver.

Mistake 7: Ignoring Governance

Data models don't exist in a vacuum. They contain PII, financial data, health records, and other sensitive information. Ignoring governance creates compliance risk and erodes trust.

Common governance gaps:

No access controls (everyone sees everything)
No classification (no one knows which columns contain PII)
No ownership (no one knows who to ask about table X)
No lineage (no one knows where the data came from)

Fix: Integrate governance from day one:

Tag columns by sensitivity (PII, financial, public)
Assign ownership per table or domain
Apply row and column-level access policies
Document data lineage from source to consumption

In Dremio, Fine-Grained Access Control enforces row and column-level policies, Labels classify datasets, and the Open Catalog tracks lineage. Governance is part of the platform, not an afterthought.

What to Do Next

Pick one of these seven mistakes. Check whether your current data model has it. Fix it. Then move to the next one. Data modeling is iterative — no team gets it perfect on the first pass. The goal is not perfection but continuous improvement: clearer names, better documentation, tighter governance, and models that match what your consumers actually need.

Try Dremio Cloud free for 30 days

Semantic Layer Best Practices: 7 Mistakes to Avoid

Alex Merced — Wed, 18 Feb 2026 18:00:00 GMT

Semantic layers don't fail because the technology is wrong. They fail because of design decisions made in the first two weeks — choices that seem reasonable at the time and create compounding problems for months afterward.

Here are the seven mistakes that kill semantic layer projects, and how to avoid each one.

Mistake 1: Defining Metrics in Multiple Places

What happens: Revenue is defined in a Tableau calculated field, a Power BI DAX measure, a dbt model, and a SQL view. Four sources of truth. None of them agree.

Why it's common: Teams adopt new tools without migrating metric definitions. Each tool gets its own model. Over time, the definitions drift.

The fix: Every metric gets exactly one canonical definition in the semantic layer. All downstream tools query that definition. No exceptions. When someone needs Revenue, they query business.revenue, not their own formula.

This principle extends to AI agents. If your AI generates its own metric formulas instead of referencing the semantic layer, you've just added another source of truth — the least trustworthy one.

Mistake 2: Skipping the Bronze Layer

What happens: A data engineer creates a Silver view that joins raw source tables directly, mixing data cleanup (type casting, column renaming) with business logic (filters, calculations) in a single query. When the source schema changes — a column is renamed, a type is modified — the Silver view breaks.

Why it's common: The Bronze layer feels redundant. It's just a 1:1 mapping of the source. Why add a layer that doesn't change anything?

The fix: The Bronze layer absorbs schema changes. When a source renames col_7 to order_date_utc, you update one Bronze view. The Silver and Gold views above it don't change. This insulation is worth the tiny overhead of maintaining passthrough views.

Bronze views also standardize data formats. Timestamps normalized to UTC. Strings cast to consistent encodings. Column names made human-readable. This cleanup happens once, at the bottom of the stack, and every view above benefits.

Mistake 3: Using SQL Reserved Words as Column Names

What happens: A Bronze view exposes a column called Date. Now every downstream query must reference "Date" with double quotes. Analysts forget. AI agents don't quote it at all. Queries break intermittently. Debugging is frustrating because the error messages are cryptic.

Why it's common: Source systems often use generic names. Date, Timestamp, Order, Group, Role — all are SQL reserved words. Bronze views that don't rename them propagate the problem to every consumer.

The fix: Rename early. In the Bronze layer, map Date to TransactionDate, Timestamp to EventTimestamp, Order to CustomerOrder. Use domain-specific prefixes that are unambiguous and never conflict with SQL keywords.

This small decision saves hundreds of hours of debugging across the life of the semantic layer. It also dramatically improves AI agent accuracy, since language models generating SQL rarely add appropriate quoting for reserved words.

Mistake 4: Building Without Stakeholder Input

What happens: A data engineering team builds 50 Silver views based on the database schema. They expose every table, every column, every possible metric. Business users look at the result, don't recognize any of the terms, and go back to their spreadsheets.

Why it's common: Data engineers understand the schema. They assume the schema structure maps to business needs. It usually doesn't.

The fix: Start with a metric glossary co-created with stakeholders from Sales, Finance, Marketing, and Product. Ask them: What are your top 5 metrics? How do you calculate them? What decisions do they drive? Build the Silver layer around those answers, not around the database schema.

This step feels slow. It's the fastest path to adoption. A semantic layer that uses business language and models business concepts gets adopted. A semantic layer that mirrors the database schema gets ignored.

Mistake 5: Treating Documentation as Optional

What happens: Views are created with no Wikis, no column descriptions, no Labels. The semantic layer works for the person who built it. Everyone else — analysts, AI agents, new team members — can't figure out what the views mean.

Why it's common: Documentation takes time. Deadlines are tight. Teams plan to "add documentation later." Later never comes.

The fix: Make documentation part of the view creation process, not a follow-up task. At minimum, every view gets:

A one-sentence description of what it represents
Labels for governance (PII, Finance, Certified)
Column descriptions for any non-obvious field

Modern platforms reduce this burden with AI-generated documentation. Dremio's generative AI samples table data and auto-generates Wiki descriptions and Label suggestions. The AI provides a 70% first draft. The data team adds domain context for the other 30%.

Undocumented views are invisible to AI agents. If the Wiki is empty, the AI agent has no context to generate accurate SQL. Documentation isn't just nice to have. It's an accuracy requirement.

Mistake 6: Applying Security at the BI Tool Level Only

What happens: Row-level security is configured in Tableau so regional managers only see their region. Then an analyst opens a SQL client, queries the underlying table directly, and sees all regions. The security was enforced in the dashboard, not in the data.

Why it's common: BI tools make it easy to apply filters and security rules. Data platforms require more setup. Teams take the easy path.

The fix: Enforce access policies at the semantic layer, not the BI layer. Row-level security and column masking should be applied on the virtual datasets (views). Every query path — dashboard, notebook, API, AI agent — inherits the same rules.

Dremio implements this through Fine-Grained Access Control (FGAC): policies defined as UDFs at the view level. A regional manager queries business.revenue and automatically sees only their region, regardless of how they access the data. No security gaps between tools.

Mistake 7: Trying to Model Everything at Once

What happens: The team commits to building a complete semantic layer covering every source, every table, and every metric. The project takes six months. By the time it launches, requirements have changed, stakeholder interest has waned, and half the views are out of date.

Why it's common: Ambitious leaders want a "complete" solution. Data teams want to avoid rework. Neither wants to ship an incomplete layer.

The fix: Start with 3-5 core metrics that the organization actively debates (usually Revenue, Active Users, Churn). Build one Bronze → Silver → Gold pipeline per metric. Validate that the same question produces the same answer across two different tools.

Once those metrics are stable, expand incrementally. Add new sources, new views, new metrics — one at a time. Each addition is low-risk because the layered architecture isolates changes. A new Gold view doesn't affect existing Silver views.

The fastest semantic layers reach 80% organizational coverage not by modeling everything up front, but by proving value quickly and expanding from momentum.

What to Do Next

Pick one mistake from this list. Check whether your semantic layer (or your plan for one) is making it. Fix that one thing this week. Then come back for the next one.

Try Dremio Cloud free for 30 days

Pipeline Observability: Know When Things Break

Alex Merced — Wed, 18 Feb 2026 17:00:00 GMT

An analyst messages you on Slack: "The revenue numbers look wrong. Is the pipeline broken?" You check the orchestrator — all green. You check the target table — data loaded this morning. You check the row count — looks normal. Forty-five minutes later, you discover that a source API returned empty responses for one region, and the pipeline happily loaded zero rows for that region without alerting anyone.

The pipeline succeeded. The data was wrong. No one knew until a human noticed.

This is the cost of monitoring pipeline execution without monitoring pipeline output.

You Can't Fix What You Can't See

Traditional monitoring answers: did the job run? Did it succeed? How long did it take? These questions cover infrastructure health, not data health. A pipeline can execute perfectly — no errors, no retries, no timeouts — and still produce incorrect or incomplete data.

Observability goes further. It answers: what did the pipeline process? How much? Was the data complete and correct? Is the output fresh? And when something is wrong, it provides enough context to diagnose the root cause without hunting through logs manually.

The distinction matters. Monitoring tells you the pipeline ran. Observability tells you the pipeline worked.

The Three Pillars of Pipeline Observability

Metrics. Quantitative measurements collected at every pipeline stage: row counts, processing time, error rates, data freshness, resource utilization. Metrics are cheap to collect, easy to aggregate, and essential for dashboards and alerting.

Logs. Structured, timestamped records of what happened during execution. A useful log entry includes: pipeline name, stage name, batch ID, timestamp, action (started/completed/failed), row count, and any error message. Structured logs (JSON format) are searchable and parseable. Unstructured logs ("Processing data...") are noise.

Lineage. The path data takes from source to destination, at the table or column level. Lineage answers: where did this number come from? If the source changes, what downstream tables and dashboards are affected? Lineage turns debugging from archaeology into graph traversal.

What to Measure

Not everything needs a metric. Measure what helps you answer these questions:

Is the data fresh? Track the timestamp of the most recent row in each target table. Compare it to the expected freshness (e.g., less than 2 hours old). A freshness metric that exceeds its SLA triggers an alert before anyone opens a dashboard.

Is the data complete? Track row counts in vs. row counts out at each stage. A significant drop (e.g., input: 100,000 rows, output: 90,000 rows) means records were filtered, rejected, or lost.

Is the data correct? Track quality metrics: null rates, duplicate rates, range violation counts. Trend these over time. A gradual increase in null rates indicates a deteriorating source.

Is the pipeline healthy? Track execution time per stage. A stage that normally takes 5 minutes but now takes 50 minutes may indicate data volume growth, resource contention, or a bad query plan.

Is the pipeline meeting SLAs? Define when data must be available (e.g., daily tables loaded by 6 AM). Track SLA compliance as a percentage. A pipeline with 95% SLA compliance has failed its consumers once every 20 days.

Alerting Without Alert Fatigue

Alert fatigue is the most common reason observability fails. Too many alerts and the on-call engineer starts ignoring them. Too few and real problems go unnoticed.

Alert on business impact, not on every error. A transient retry is not an alert. A pipeline that misses its SLA by an hour is. A single null row is not an alert. A null rate jumping from 0.1% to 15% is.

Use severity levels. Critical: data consumers are affected now (missed SLA, empty output). Warning: something is degrading but not yet impacting consumers (execution time increasing, row count declining). Info: notable but non-actionable (successful backfill, schema migration completed).

Set thresholds dynamically. Static thresholds ("alert if row count < 10,000") break when data naturally grows or shrinks. Use rolling baselines: alert if today's row count deviates by more than 20% from the 7-day average.

Route alerts effectively. Critical alerts go to PagerDuty or on-call channels. Warnings go to team Slack channels. Info goes to logs-only. Don't send everything to the same channel.

Data Lineage for Impact Analysis

When a problem occurs, the first question is: what's affected? Lineage answers this.

Upstream analysis. A dashboard shows wrong numbers. Lineage traces the dashboard back through the serving table, the transformation, the staging table, and the raw source. The break is visible in the graph.

Downstream impact analysis. A source system announces a schema change. Lineage shows every table, model, and dashboard that depends on that source. You know the blast radius before making any changes.

Column-level lineage. Table-level lineage shows connections between tables. Column-level lineage shows which source column feeds which target column. This level of detail turns a "the revenue is wrong" investigation from hours to minutes.

What to Do Next

Add freshness tracking to your three most critical tables: record the max event timestamp after each load and alert when it exceeds the SLA. This single metric — data freshness — catches more problems than any other observability signal.

Try Dremio Cloud free for 30 days

Data Vault Modeling: Hubs, Links, and Satellites

Alex Merced — Wed, 18 Feb 2026 17:00:00 GMT

Dimensional modeling works well when your source systems are stable and your business questions are predictable. But what happens when sources change constantly, new systems get added every quarter, and regulatory requirements demand a full audit trail of every attribute change?

Data Vault modeling was designed for exactly this scenario. Created by Dan Linstedt, it separates data into three distinct table types — Hubs, Links, and Satellites — each handling a different concern: identity, relationships, and descriptive context.

What Problem Data Vault Solves

Traditional dimensional models embed everything about a business entity in one dimension table. A dim_customers table contains the customer ID, name, address, segment, acquisition channel, and lifetime value. When a new source system provides additional customer attributes, you add columns to dim_customers. When business rules change how "segment" is calculated, you update the ETL pipeline that populates that table.

Over time, these dimension tables become fragile. They depend on multiple source systems. A change in one source breaks the ETL. Schema changes require coordinated updates across pipelines, tables, and downstream reports.

Data Vault solves this by decomposing entities into independent components that can evolve separately.

The Three Building Blocks

Hubs: Business Identity

A Hub stores unique business keys — the identifiers that define a business entity regardless of which source system provides them.

CREATE TABLE hub_customer (
    customer_hash_key BINARY(32),  -- Hash of the business key
    customer_id VARCHAR(50),        -- Natural business key
    load_date TIMESTAMP,
    record_source VARCHAR(100)
);

Hubs are immutable. Once a business key is loaded, it never changes. A customer who has customer_id = 'C-1042' always has that key. Hubs answer the question: What business concepts exist?

Links: Relationships

A Link stores relationships between Hubs. Every relationship — customer-to-order, order-to-product, employee-to-department — gets its own Link table.

CREATE TABLE link_customer_order (
    link_hash_key BINARY(32),
    customer_hash_key BINARY(32),
    order_hash_key BINARY(32),
    load_date TIMESTAMP,
    record_source VARCHAR(100)
);

Links are also immutable. Once a relationship is recorded, it stays. Links support many-to-many relationships by default. They answer the question: How are business concepts related?

Satellites: Descriptive Context

Satellites store the descriptive attributes of a Hub or Link, along with their change history.

CREATE TABLE sat_customer_details (
    customer_hash_key BINARY(32),
    effective_date TIMESTAMP,
    customer_name VARCHAR(200),
    email VARCHAR(200),
    city VARCHAR(100),
    segment VARCHAR(50),
    load_date TIMESTAMP,
    record_source VARCHAR(100)
);

Every time an attribute changes, a new Satellite row is inserted. This is equivalent to SCD Type 2 — full history is preserved without modifying existing rows. Different source systems can feed different Satellites for the same Hub, allowing attributes to arrive independently.

How a Data Vault Query Works

To reconstruct a business entity (like a current customer profile), you join the Hub to its current Satellite rows:

SELECT
    h.customer_id,
    s.customer_name,
    s.email,
    s.city,
    s.segment
FROM hub_customer h
JOIN sat_customer_details s ON h.customer_hash_key = s.customer_hash_key
WHERE s.effective_date = (
    SELECT MAX(effective_date)
    FROM sat_customer_details s2
    WHERE s2.customer_hash_key = s.customer_hash_key
);

This is more complex than querying dim_customers directly. That complexity is the primary criticism of Data Vault. In practice, teams build a presentation layer — star schema views on top of the vault — for business users and BI tools.

Platforms like Dremio make this practical. The raw vault tables live in the Bronze layer. Silver-layer views reconstruct business entities by joining Hubs, Links, and Satellites. Gold-layer views present dimensional star schemas for dashboards and AI agents. Users never query the vault tables directly.

When Data Vault Fits

Multiple source systems that change frequently. Adding a new source means adding new Satellites — not redesigning existing tables. The Hub and Link structure remains stable.

Regulated industries requiring full audit trails. Financial services, healthcare, and government often need to prove what data looked like at any point in time. Satellites provide that out of the box.

Large enterprises with parallel development teams. Hubs, Links, and Satellites can be loaded independently, enabling parallel ETL development without pipeline conflicts.

Long-term data warehouses with decades of history. The separation of structure (Hubs, Links) from content (Satellites) makes the vault resilient to business changes over time.

When Data Vault Doesn't Fit

Small teams or simple source environments. If you have five source tables and one BI tool, Data Vault adds complexity without proportional benefit. A star schema is faster to build and easier to maintain.

Direct BI tool access. BI tools don't speak Data Vault natively. You always need a presentation layer on top, which means building two models instead of one.

Speed-to-value projects. When the goal is "get a dashboard live this sprint," Data Vault's up-front design work slows you down.

Factor	Data Vault	Dimensional Model
Source flexibility	High	Moderate
Audit trail	Built-in	Optional (SCDs)
Query simplicity	Low (needs presentation layer)	High
Learning curve	High	Moderate
Adding new sources	Easy (new satellites)	Harder (redesign dimensions)
BI tool compatibility	Low	High

What to Do Next

If you're evaluating Data Vault, start by counting your source systems and estimating how often they change schema. If the answer is "more than five sources" and "at least once a quarter," Data Vault's separation of concerns will likely save you from painful redesign cycles. If your environment is simpler than that, a well-designed dimensional model will get you to production faster.

Try Dremio Cloud free for 30 days

How a Self-Documenting Semantic Layer Reduces Data Team Toil

Alex Merced — Wed, 18 Feb 2026 17:00:00 GMT

Every data team knows documentation is important. And almost every data team has a backlog of undocumented tables, unlabeled columns, and outdated descriptions that nobody has time to fix. The problem isn't motivation. It's that manual documentation doesn't scale.

A self-documenting semantic layer changes the equation. Instead of asking humans to describe every column in every table, the platform generates descriptions automatically, suggests governance labels from data patterns, and propagates context through the view chain. Documentation becomes a byproduct of building the semantic layer, not a separate project.

The Documentation Problem Nobody Solves

Industry surveys consistently find that 70% or more of enterprise data assets are undocumented or poorly documented. The result: analysts spend 30-40% of their time searching for data and trying to understand what it means before they can start analyzing it.

This isn't just a productivity problem. Undocumented data is a governance risk. A column named status with values 0, 1, 2, and 3 could mean anything. An analyst guesses. An AI agent guesses worse. Nobody verifies. The wrong assumptions get baked into dashboards that drive business decisions.

Data teams respond with documentation sprints. They burn a week writing Wiki pages for their top 50 tables. Two months later, half the descriptions are outdated because schemas have changed. The cycle repeats.

What Self-Documenting Actually Means

A self-documenting semantic layer generates and maintains documentation with minimal human effort. Three mechanisms work together:

AI-generated descriptions: The platform samples data in a table and generates human-readable descriptions for each column and the table itself.

Automated label suggestions: The platform analyzes column names, data types, and value patterns to suggest governance labels (PII, Finance, Certified).

Metadata propagation: When a Silver view references a Bronze view, column descriptions flow downstream automatically. Documentation written once at the Bronze level appears everywhere the column is used.

Human oversight is still essential. AI provides a 70% first draft. Data engineers add the domain-specific context that only they know: business rules, edge cases, known data quality issues. The point isn't to eliminate human documentation. It's to eliminate the blank page.

AI-Generated Descriptions

Modern semantic layer platforms can sample a table's data and generate meaningful descriptions automatically.

Consider a column named cltv in a table called customers. The AI samples values (1200.50, 3400.00, 780.25), examines the column name and table context, and generates:

cltv: Customer Lifetime Value in USD. Represents the total revenue attributed to this customer from their first purchase to the current date, excluding refunded transactions.

Not every generated description will be this precise. But most are useful enough to replace the current state: an empty description that tells the analyst nothing.

More examples:

A column with values "US", "UK", "DE" → "ISO 3166 alpha-2 country code for the customer's billing address"
A DATE column named created_at in a subscriptions table → "Date the subscription was created"
A FLOAT column named mrr → "Monthly Recurring Revenue in the account's base currency"

Automated Label Suggestions

Labels categorize data for governance and discovery. Manually tagging every column in a data warehouse with hundreds of tables is impractical. AI-based label suggestion makes it manageable:

Columns containing email-like patterns (text with @ symbols) → suggested label: PII
Columns with phone number patterns → suggested label: PII
Columns named price, total, amount, revenue → suggested label: Finance
Columns in tables marked "Certified" → suggested label propagated to downstream views

Dremio's approach combines these suggestions with human approval. The AI proposes labels. A data engineer reviews and accepts or rejects. Over time, the catalog fills up with accurate, useful labels without dedicated labeling sprints.

Metadata Propagation Through Views

In a well-designed semantic layer, documentation shouldn't need to be written more than once. The Bronze-Silver-Gold view architecture creates a natural propagation path:

Bronze layer: Document the CustomerID column as "Unique identifier for the customer, sourced from the CRM system."
Silver layer: A Silver view references CustomerID. The description propagates automatically. No re-documentation needed.
Gold layer: An aggregated Gold view groups by CustomerID. The description carries through.

This propagation is especially valuable for join columns, filter columns, and commonly used dimensions that appear in dozens of views. Write the description once at the source, and it follows the column everywhere.

How This Reduces Toil

The impact on data team productivity is measurable:

Documentation Task	Manual Approach	Self-Documenting
Column descriptions	Write each by hand	AI generates draft, human refines
Governance labels	Manual tagging sprint	AI suggests from data patterns
Downstream view docs	Re-write for each view	Propagated from upstream
Schema change updates	Manually check and update	AI re-scans and flags changes
New table onboarding	Create from scratch	AI generates baseline immediately

The net effect: documentation coverage goes from 30% (what the team could manage manually) to 80-90% (AI baseline + human refinement). The team spends hours instead of weeks on documentation. And the documentation stays current because the AI can re-scan when schemas change — flagging outdated descriptions instead of waiting for someone to notice.

For AI agents, this improvement is material. A richer, more accurate semantic layer means the AI generates better SQL, hallucinates less, and requires fewer corrections. Self-documentation isn't just a productivity feature. It's an AI accuracy feature.

What to Do Next

Pick your most-used table. Open it in your data platform. How many columns have descriptions? How many have governance labels? If the answer is "not many," calculate how long it would take to document the entire table manually. Then consider a platform that does 70% of that work for you.

Try Dremio Cloud free for 30 days

Testing Data Pipelines: What to Validate and When

Alex Merced — Wed, 18 Feb 2026 16:00:00 GMT

Ask an application developer how they test their code and they'll describe unit tests, integration tests, CI/CD pipelines, and coverage metrics. Ask a data engineer the same question and the most common answer is: "we check the dashboard."

Data pipelines are software. They have inputs, logic, and outputs. They can have bugs. They can break silently. And unlike application bugs that trigger error pages, data bugs produce numbers that look plausible — until someone makes a business decision based on them.

Pipelines Are Software — They Need Tests

The bar for data pipeline testing shouldn't be lower than for application code. If anything, it should be higher. Application bugs are usually visible (broken UI, failed request). Data bugs are invisible (wrong aggregation, missing rows, stale values) and their impact compounds over time.

Yet most data teams have no automated tests. They rely on manual spot-checks, analyst complaints, and hope. Testing a pipeline means catching problems before they reach consumers, not after.

The Testing Pyramid for Data

Borrow the testing pyramid from software engineering and adapt it for data:

Base: Schema and contract tests. Fast, cheap, run on every pipeline execution. Does the output schema match what consumers expect? Do required columns exist? Are data types correct? These tests catch structural problems (dropped columns, type changes) immediately.

Middle: Data validation tests. Check the values in the output. Are primary keys unique? Are required columns non-null? Do amounts, dates, and counts fall within valid ranges? These tests catch quality problems (duplicates, nulls, outliers) before they propagate.

Top: Regression and integration tests. Compare today's output to historical patterns. Did the row count change dramatically? Did the total revenue shift by more than 10%? These tests catch subtle logic errors and upstream data changes.

Run more tests at the base (they're cheap and fast) and fewer at the top (they're expensive but comprehensive).

Schema and Contract Tests

Schema tests are the simplest and most impactful place to start. After every pipeline run, verify:

Column existence. Every expected column is present in the output. If a transformation accidentally drops a column, you want to know immediately — not when a downstream query fails.

Data types. Columns have their expected types. A revenue column that silently became a string will pass a NULL check but break calculations.

Not-null constraints. Required columns contain no nulls. An order table where customer_id is null means the join to the customer table will silently lose rows.

Uniqueness. Primary key columns have no duplicates. Duplicate order IDs mean double-counted revenue.

-- Example schema and contract tests
-- Check for unexpected nulls
SELECT COUNT(*) AS null_count
FROM orders
WHERE order_id IS NULL OR customer_id IS NULL;

-- Check for duplicates
SELECT order_id, COUNT(*) AS cnt
FROM orders
GROUP BY order_id
HAVING COUNT(*) > 1;

Runtime Data Validation

Schema tests verify structure. Data validation tests verify content. Run these after every pipeline execution, before marking the job as successful:

Range checks. Numeric values fall within expected bounds. An order total of -$500 or $999,999,999 is likely a bug. Define acceptable ranges per column and flag outliers.

Referential integrity. Foreign keys reference existing records. An order with product_id = 12345 should correspond to a row in the products table. Missing references indicate either missing data or a pipeline timing issue.

Freshness checks. The most recent event timestamp is within the expected window. If a daily pipeline's output contains no events from today, something went wrong — even if the job succeeded.

Volume checks. Row counts fall within historical norms. A daily feed that normally produces 50,000 rows but arrives with 500 should trigger an alert. Use percentage thresholds (±20% from the trailing 7-day average) to avoid false positives.

Custom business rules. Domain-specific assertions. "Every invoice must have at least one line item." "No employee should have a start date in the future." These rules encode business knowledge that generic tests can't capture.

Regression and Anomaly Detection

Regression tests compare today's output to historical baselines:

Aggregate comparison. Compare key metrics (total revenue, row count, distinct customer count) against the previous run. Deviations beyond a threshold (e.g., ±15%) may indicate an upstream change, a bug in new transformation logic, or missing source data.

Distribution checks. Compare the distribution of categorical columns (status values, country codes) against historical norms. A sudden spike in "unknown" status may indicate a schema change in the source.

Trend analysis. Track metrics over time. A gradual decline in row count over weeks may indicate a leak that daily checks miss.

Regression tests are more expensive to maintain because they require historical baselines and threshold tuning. Start simple (row count ± 20%) and refine as you learn what normal looks like.

What to Do Next

Add three tests to your most critical pipeline today: a uniqueness check on the primary key, a null check on required columns, and a row count comparison against yesterday's output. Run them after every pipeline execution. These three tests alone will catch the majority of data problems before they reach consumers.

Try Dremio Cloud free for 30 days

Denormalization: When and Why to Flatten Your Data

Alex Merced — Wed, 18 Feb 2026 16:00:00 GMT

Normalization is the first rule taught in database design. Eliminate redundancy. Store each fact once. Use foreign keys. It's the right rule for transactional systems. And it's the wrong rule for most analytics workloads.

Denormalization is the deliberate introduction of redundancy into your data model to reduce joins and speed up queries. Done poorly, it creates a maintenance nightmare. Done well, it turns slow dashboards into fast ones and makes your data accessible to analysts and AI agents who can't write 12-table joins.

What Normalization Gives You (and What It Costs)

Normalization (Third Normal Form and beyond) organizes data so that each piece of information exists in exactly one place. A customer's city lives in the customers table. An order's product lives in the order_items table joined to the products table.

What normalization gives you:

No update anomalies (change a city in one row, not thousands)
Smaller storage footprint (no duplicated data)
Strong data integrity (constraints enforced at the schema level)

What normalization costs you:

More joins per query (a report might join 10-15 tables)
Slower read performance (each join adds latency)
More complex SQL (longer queries, more error-prone)
Harder self-service (analysts struggle with multi-join queries)

For an OLTP system processing 10,000 inserts per second, normalization is correct. For an OLAP system answering "revenue by region by quarter," it's a performance bottleneck.

What Denormalization Actually Means

Denormalization takes several forms:

Embedding dimension attributes in fact tables. Instead of joining orders → customers to get the customer name, include customer_name directly in the orders table.

Pre-joining lookup tables. Instead of maintaining separate cities, states, and countries tables, flatten them into a single column: customer_city_state_country.

Adding calculated columns. Instead of computing quantity × price × (1 - discount) in every query, store net_revenue as a pre-computed column.

Creating wide summary tables. Instead of joining across 8 tables for a monthly report, create a monthly_summary table with all needed columns in one place.

The key insight: denormalization trades write-time simplicity for read-time simplicity. Updating a customer's city now requires updating it in multiple places. But querying revenue by city no longer requires a join.

When to Denormalize

Analytics and reporting workloads. If your model primarily serves dashboards, reports, and ad-hoc queries, denormalization reduces query time and complexity.

Self-service environments. Business users selecting fields in a BI tool get better results from a wide, flat table than from a web of normalized tables they don't understand.

AI-driven queries. When an AI agent generates SQL, fewer tables and fewer joins reduce the chance of wrong join conditions and hallucinated relationships.

Read-heavy, write-light patterns. If your data loads once a day (batch ETL) and gets queried thousands of times, optimizing for reads makes sense.

When NOT to Denormalize

High-frequency transactional writes. If your system processes real-time inserts and updates, denormalized redundancy creates update anomalies. A customer moving to a new city means updating hundreds of order rows.

When consistency matters more than speed. Financial systems with audit requirements often need the strict integrity that normalization provides.

Small datasets. If the query joins 5 tables with 1,000 rows each, denormalization won't improve performance noticeably. The overhead of redundancy isn't worth the marginal speed gain.

The Tradeoffs

Benefit	Cost
Fewer joins per query	Update anomalies (same data in multiple places)
Faster read performance	Larger storage footprint
Simpler SQL for analysts	Pipeline complexity (keeping redundant data in sync)
Better BI tool compatibility	Risk of inconsistency if pipelines fail
AI agents write more accurate SQL	More effort to maintain data quality

Virtual Denormalization: The Middle Path

There's a way to get the query benefits of denormalization without the physical redundancy: SQL views.

A view can join and flatten multiple normalized tables into a single logical table. Consumers query the view as if it's one wide table — simple SQL, no joins required. But the underlying data stays normalized. Update a customer's city in the customers table, and the view reflects the change automatically.

CREATE VIEW v_orders_enriched AS
SELECT
    o.order_id,
    o.order_date,
    c.customer_name,
    c.city AS customer_city,
    p.product_name,
    p.category AS product_category,
    o.quantity * o.unit_price AS revenue
FROM orders o
JOIN customers c ON o.customer_id = c.customer_id
JOIN products p ON o.product_id = p.product_id;

Analysts query v_orders_enriched without knowing the underlying structure. The join logic is defined once and reused by everyone.

The tradeoff: views execute the joins at query time. For very large datasets, this can be slow. Platforms like Dremio solve this with Reflections — which physically materialize the view's results in an optimized format, updated automatically. Users still query the logical view, but the engine substitutes the pre-computed Reflection for performance. You get the simplicity of denormalization, the consistency of normalization, and the speed of materialization.

What to Do Next

Identify your most-queried report or dashboard. Count the joins in the underlying SQL. If there are more than five, create a denormalized view that flattens the data. Compare query performance before and after. If the view is still too slow for your SLA, adding a materialized acceleration layer (like Reflections) closes the gap.

Try Dremio Cloud free for 30 days

Headless BI: How a Universal Semantic Layer Replaces Tool-Specific Models

Alex Merced — Wed, 18 Feb 2026 16:00:00 GMT

Your organization uses Tableau for executive dashboards, Power BI for operational reports, and Python notebooks for data science. Revenue is defined in Tableau's calculated field, Power BI's DAX measure, and a SQL query inside a Jupyter notebook. Three tools. Three definitions. None of them match.

This is what happens when semantic models are locked inside BI tools. Headless BI fixes it by pulling the definitions out.

The Problem with Tool-Specific Semantic Models

Every major BI tool comes with its own modeling layer. Looker has LookML. Tableau has the Data Model. Power BI has DAX and the tabular model. Each one defines metrics, relationships, and calculated fields in a proprietary format.

This creates three problems:

Definition duplication. Every metric must be defined in every tool. Revenue in Tableau. Revenue in Power BI. Revenue in the data science notebook. When the formula changes (say, a new exclusion rule is added), you update it in three places. Or you forget one, and your dashboards disagree.

Tool lock-in. Your metric definitions are trapped inside the tool's proprietary format. Switching from Tableau to a different visualization layer means rebuilding every metric from scratch. The data model doesn't migrate.

AI agent exclusion. When you add an AI agent to your stack, it can't access the Looker LookML definitions or the Power BI DAX measures. It has no semantic model to work with. It generates SQL based on raw table schemas and gets the formulas wrong.

What Headless BI Means

Headless BI is an architecture pattern where metric definitions and business logic are decoupled from the visualization layer. The "head" (the dashboard or chart) is separate from the "body" (the semantic definitions).

In a headless architecture:

Metrics are defined once in a platform-neutral semantic layer
Definitions are exposed via standard interfaces: SQL, JDBC, ODBC, Arrow Flight, REST
Any tool — Tableau, Power BI, Python, an AI agent, a custom app — connects to the same definitions
Adding a new visualization tool requires zero metric migration

The semantic layer becomes a shared service. Visualization tools consume it. They don't own it.

Tool-Specific vs. Universal Semantic Layer

Dimension	Tool-Specific Model	Universal Semantic Layer
Where metrics are defined	Inside each BI tool	Centralized, tool-independent
Number of Revenue definitions	One per tool	One total
Formula change process	Update every tool	Update once, propagates
New tool onboarding	Rebuild all definitions	Connect and query
AI agent access	No (locked in BI format)	Yes (standard SQL interface)
Portability	Vendor-locked	Open and interoperable

What Composable Analytics Looks Like

Headless BI is one piece of a broader shift called composable analytics. Instead of buying a monolithic BI platform that bundles data modeling, metric definitions, and visualizations together, you assemble your analytics stack from modular, interchangeable components.

The semantic layer is the metric module. Choose any visualization tool on top. Choose any data storage underneath. Swap components without rebuilding definitions.

This modularity matters most for AI. An AI agent becomes a first-class consumer of the semantic layer, alongside dashboards and notebooks. It connects to the same interface, reads the same metric definitions, and gets the same answers. No special integration needed.

How This Works in Practice

Dremio functions as a universal semantic layer that any tool can consume. The architecture:

Virtual datasets (SQL views) define business logic and metric calculations once
Wikis and Labels document business context for human and AI consumers
Fine-Grained Access Control enforces security policies at the query level
Reflections optimize performance automatically for any consumer

Connection options include ODBC, JDBC, Arrow Flight (for columnar high-speed clients), and REST API. A Tableau dashboard connects via ODBC. A Python notebook connects via Arrow Flight. Dremio's AI Agent reads the Wikis and Labels to generate accurate SQL from natural language. All three hit the same virtual datasets. All three get the same answers.

Because the entire semantic layer is built on open standards (Apache Iceberg for data, Apache Polaris for the catalog), the definitions aren't locked to Dremio's format. You can inspect, export, and query the same data with any Iceberg-compatible engine.

What to Do Next

Count the number of places your organization defines its top metric (probably Revenue or Monthly Active Users). If that number is greater than one, you're paying a consistency tax every time someone changes the formula. A universal semantic layer eliminates that tax by defining it once and serving it everywhere.

Try Dremio Cloud free for 30 days

Partition and Organize Data for Performance

Alex Merced — Wed, 18 Feb 2026 15:00:00 GMT

A table with 500 million rows takes 45 seconds to query. After partitioning it by date, the same query — filtering on a single day — returns in 2 seconds. The SQL didn't change. The data didn't change. The only thing that changed was how the data was organized on disk.

Performance in analytical workloads is almost never about faster hardware. It's about reading less data.

Read Less Data, Run Faster Queries

Analytical query engines scan data to answer queries. A full table scan reads every row, every column. But most queries only need a fraction of the data: this week's transactions, this region's customers, this product category's sales.

Partitioning and data organization let the engine skip irrelevant data entirely:

Partition pruning. The engine reads only the partitions that match the query's WHERE clause.
Column pruning. Columnar formats (Parquet, ORC) read only the requested columns.
Predicate pushdown. Min/max statistics in file metadata let the engine skip files whose value ranges don't match the filter.

Combined, these techniques can reduce the data scanned from terabytes to megabytes. The fastest query is the one that reads the least data.

Partitioning Strategies

Time-based partitioning. Partition by date, hour, or month. This is the most common strategy because most analytical queries filter by time. A daily partition structure means a query for "last week" reads 7 partitions instead of scanning the entire table.

Value-based partitioning. Partition by a categorical column: region, source system, customer tier. This works when queries consistently filter on that column. A multi-tenant application might partition by tenant ID so each tenant's queries touch only their data.

Hash-based partitioning. Distribute data evenly across N buckets using a hash function on a key column. This is useful for join-heavy workloads: two tables hashed on the same join key can be joined partition-to-partition without shuffling data.

Composite partitioning. Combine strategies: partition by date, then bucket by customer ID within each date. This handles queries that filter on date and join on customer ID.

Choosing the right strategy: Look at your most frequent queries. What columns appear in WHERE clauses and JOIN conditions? Those are your partition candidates. If 90% of queries filter by date, partition by date.

File-Level Organization

Partitioning controls which directory the query engine reads. File-level organization controls how efficiently it reads within that directory.

Sorting. Sort rows within each file by a frequently filtered column. If queries often filter WHERE status = 'active', sorting by status clusters active rows together. The engine reads min/max metadata, sees that a file's status range is only 'active', and skips files that don't match.

Z-ordering. When queries filter on multiple columns, linear sorting optimizes for only one. Z-ordering interleaves the sort order across multiple columns, enabling predicate pushdown on any combination of the Z-ordered columns. It's especially effective for 2-3 column filter combinations.

File sizing. Target file sizes between 128 MB and 1 GB. Files too small (< 10 MB) create metadata overhead and excessive file-open operations. Files too large (> 2 GB) reduce parallelism and waste I/O when only a fraction of the file is needed.

Compaction: The Maintenance Task You Can't Skip

Streaming writes and frequent small batch appends create many small files. A partition with 10,000 files of 1 MB each is dramatically slower to query than the same data in 10 files of 1 GB each.

Compaction merges small files into optimally-sized files. It's the data equivalent of defragmenting a disk.

Run compaction:

After streaming writes accumulate small files
After many small batch appends
On a regular schedule (daily or weekly) for active partitions
Targeted at partitions where file counts exceed a threshold

Compaction also provides an opportunity to re-sort data within files, clean up deleted records (in formats that use soft deletes like Iceberg and Delta), and update file-level statistics.

Common Partitioning Mistakes

Over-partitioning. Partitioning by a high-cardinality column (user ID, transaction ID) creates millions of partitions, each with a few rows. The engine spends more time listing and opening files than reading data. Rule of thumb: keep individual partition sizes above 100 MB.

Under-partitioning. A single partition for the entire table means every query scans everything. If your table has billions of rows and no partitions, even simple queries are slow.

Misaligned partitions. Partitioning by month when every query filters by day means the engine reads an entire month's data for a single-day query. Align partition granularity with query granularity.

Ignoring compaction. Streaming into a table without compacting creates the small-file problem. Query performance degrades gradually until someone notices. Schedule compaction as part of pipeline maintenance.

What to Do Next

Identify your slowest analytical query. Check the table's partitioning strategy. If the table has no partitions, add one aligned with the query's most common WHERE clause. If it's already partitioned, check file sizes — if the average file is under 10 MB, run compaction. Measure before and after.

Try Dremio Cloud free for 30 days

Data Modeling for Analytics: Optimize for Queries, Not Transactions

Alex Merced — Wed, 18 Feb 2026 15:00:00 GMT

The data model that runs your production application is almost never the right model for analytics. Transactional systems are designed for fast writes — inserting orders, updating inventory, processing payments. Analytics systems are designed for fast reads — scanning millions of rows, aggregating across dimensions, filtering by date ranges.

Using a transactional model for analytics is like using a filing cabinet when you need a search engine. The data is there, but finding answers takes too long.

Transactions vs. Analytics: Two Different Problems

Transactional (OLTP) workloads process many small operations: insert one order, update one account balance, delete one expired session. These models are normalized to Third Normal Form (3NF) or beyond — every piece of data stored once, redundancy eliminated, consistency enforced through constraints.

Analytical (OLAP) workloads process few large operations: scan all orders for the last year, aggregate revenue by region and product category, calculate year-over-year growth. These models are denormalized — data is pre-joined, attributes are flattened, and the structure is optimized for scans rather than updates.

Aspect	OLTP Model	OLAP Model
Optimization target	Write speed	Read speed
Normalization	3NF or higher	Denormalized
Table structure	Narrow and many	Wide and few
Joins per query	Many (10-20)	Few (3-5)
Storage format	Row-oriented	Columnar
Typical query	UPDATE one row	SUM across millions

Why Normalized Models Slow Down Analytics

A normalized 3NF model might have 15 tables involved in answering "What was revenue by product category by month?" The query engine must join orders to order_items to products to categories to dates, applying filters and aggregations across each join.

Each join adds latency. Each join also adds a point of failure — wrong join condition, missing foreign key, ambiguous column name. An AI agent generating SQL against a 15-table normalized model has far more opportunities to make a mistake than against a 4-table star schema.

The fix is not to abandon normalization. Keep your OLTP model normalized for your application. But create a separate analytical model — denormalized, structured for queries, with pre-built joins and business-friendly column names — for reporting and analytics.

Designing for Read Performance

Analytical data models follow several patterns that optimize for read performance:

Wide tables reduce joins. Instead of orders → customers → addresses → cities → states, create a single fact_orders view with customer_name, customer_city, customer_state included. Every join you eliminate saves query time and reduces complexity.

Pre-computed columns reduce repeated calculations. If every report calculates quantity * unit_price * (1 - discount) as "net revenue," compute it once in the model and expose it as a column. This eliminates repeated formula definitions and ensures consistency.

Consistent naming improves discoverability. Use order_date instead of dt. Use customer_email instead of email. When column names are self-explanatory, analysts find the right data faster, and AI agents generate more accurate SQL.

Date dimensions enable time-based analysis. A date dimension with fiscal_quarter, is_weekend, is_holiday, and week_of_year makes time-based filtering trivial. Without it, every analyst writes a different CASE WHEN MONTH(date) IN (1,2,3) THEN 'Q1' expression.

Pre-Aggregation and Summary Tables

Not every query needs to scan raw data. For frequently run aggregations, pre-aggregated summary tables reduce query time from minutes to milliseconds.

Common patterns:

Daily summary: Total revenue, order count, average order value per day per product category
Monthly snapshot: Active customers, churned customers, MRR per segment
Rolling window: 7-day and 30-day moving averages for key metrics

The tradeoff is maintenance. Every summary table needs a refresh pipeline, and stale summaries produce outdated numbers.

Platforms like Dremio handle this automatically with Reflections — pre-computed aggregations and materializations that the query optimizer uses transparently. Users query the logical views; Dremio substitutes the fastest Reflection without the user knowing. No manual summary table management required.

Columnar Storage and Physical Layout

Analytics models benefit from columnar storage formats like Parquet:

Column pruning: Queries that touch 5 of 50 columns only read those 5 columns from disk
Compression: Repeated values in a column (category names, status codes) compress efficiently
Vectorized processing: Engines like Dremio (built on Apache Arrow) process columnar data in CPU-cache-friendly batches

Physical layout decisions that matter:

Partition by time: Most analytics queries filter by date range. Partitioning by month or day lets the engine skip irrelevant data files entirely.
Sort by high-cardinality filters: If queries frequently filter by customer_id or region, sorting data within partitions enables min/max pruning.
Compact regularly: Small files from streaming inserts slow down scan performance. Compaction rewrites small files into larger, optimized ones.

What to Do Next

Find your slowest dashboard. Look at the queries behind it. Count the joins, measure the scan size, and check whether the model is normalized 3NF or denormalized for analytics. If it's still using the transactional model, create an analytical view layer on top — a denormalized star schema with pre-computed columns, clear naming, and a date dimension. The dashboard performance improvement is usually immediate and significant.

Try Dremio Cloud free for 30 days

Data Virtualization and the Semantic Layer: Query Without Copying

Alex Merced — Wed, 18 Feb 2026 15:00:00 GMT

Every data pipeline you build to move data from one system to another costs you three things: time to build it, money to run it, and freshness you lose while waiting for the next sync. Most analytics architectures accept this cost as unavoidable. It isn't.

Data virtualization eliminates the movement. A semantic layer adds meaning and governance on top. Together, they give you a complete analytics layer over distributed data without copying a single table.

The Data Movement Tax

Traditional analytics architecture looks like this: data lives in operational databases, SaaS tools, and cloud storage. To analyze it, you extract it, transform it, and load it into a central warehouse. Every source gets an ETL pipeline. Every pipeline needs monitoring, error handling, and scheduling.

The result: your analytics are always behind your operational data. The warehouse reflects what happened as of the last sync, not what's happening now. You pay for storage in both the source and the warehouse. And when you add a new source, you add a new pipeline.

This model made sense when compute was expensive and storage was local. In a cloud-native world where compute is elastic and storage is cheap, the calculus changes.

What Data Virtualization Does

Data virtualization lets you query data where it lives. Instead of copying data to a central location, you connect to each source and issue queries directly. A virtualization engine translates your SQL into the source's native protocol (JDBC for databases, S3 API for object storage, REST for SaaS), retrieves the data, and combines results from multiple sources into a single result set.

From the user's perspective, all data appears in one unified namespace. A PostgreSQL production database, an S3 data lake full of Parquet files, and a Snowflake analytics warehouse all look like tables in the same catalog.

The keyword is "no replication." The data stays where it is. The queries go to the data, not the other way around.

What a Semantic Layer Adds on Top

Virtualization solves the access problem. But access without context is dangerous. Raw access to 50 federated sources means 50 sources where analysts can write conflicting metric formulas, join tables incorrectly, and query sensitive columns without authorization.

A semantic layer added on top of virtualization provides:

Metric definitions: "Revenue" is calculated the same way regardless of which source the data comes from
Documentation: Wikis describe what each federated table and column represent in business terms
Join paths: Pre-defined relationships prevent analysts from guessing how tables connect
Access policies: Row-level security and column masking enforced at the view level, even for sources that have no fine-grained access controls of their own

The combination is powerful: you get real-time access to all your data (virtualization) with consistent meaning and governance (semantic layer), and without data movement (no ETL).

Why They're Stronger Together

Each technology is useful alone. Together, they cover gaps neither can fill individually:

Capability	Virtualization Only	Semantic Layer Only	Both Together
Access distributed data	Yes	No (limited to centralized data)	Yes
Business definitions	No	Yes	Yes
Governance enforcement	No	Yes	Yes
Zero data movement	Yes	No	Yes
Real-time access	Yes	Depends on data freshness	Yes
Unified namespace	Yes	Yes	Yes

Virtualization without a semantic layer gives you raw SQL access to everything. Powerful for engineers. Risky for an organization. No metric consistency, no governance, no documentation.

A semantic layer without virtualization covers only the data that's been moved to the platform's native storage. Every source that hasn't been ETL'd is invisible to the layer. You get great governance over a subset of your data, and no governance over the rest.

How It Works in Practice

Dremio is built on this architecture natively. It combines a high-performance virtualization engine (supporting 30+ source types including S3, ADLS, PostgreSQL, MySQL, MongoDB, Snowflake, and Redshift) with a full semantic layer (virtual datasets, Wikis, Labels, Fine-Grained Access Control).

A practical query flow:

An analyst queries business.revenue_by_region — a virtual dataset (view)
Dremio's optimizer determines that this view joins data from PostgreSQL (customer records) and S3/Iceberg (order transactions)
Predicate pushdowns push filter logic to each source (e.g., date range filters applied at the source)
Results are combined using Apache Arrow's columnar format (zero serialization overhead)
Row-level security filters the results based on the analyst's role
If a Reflection (pre-computed copy) exists, Dremio substitutes it transparently for faster performance

The analyst sees one table. Behind it, two sources, one semantic layer, and automatic performance optimization.

When to Virtualize vs. When to Materialize

Not every query should hit the source directly. The right architecture uses both strategies:

Virtualize when:

The data changes frequently and freshness matters
The dataset is queried infrequently (monthly reports, ad-hoc exploration)
Compliance requires data to stay in its source system
You're evaluating a new source before committing to a pipeline

Materialize when:

Multiple dashboards query the same dataset hundreds of times daily
Joins across sources are slow because of network latency
Table-level optimizations (compaction, partitioning, clustering) would improve performance
AI workloads need scan-heavy access to large datasets

The practical strategy: start every source as a federated (virtual) connection. Monitor query frequency and performance. When a dataset crosses the line into "queried daily by multiple teams," materialize it as an Apache Iceberg table. Dremio's Reflections automate this for the most common query patterns, creating materialized copies that the optimizer uses transparently.

What to Do Next

Count your current ETL pipelines. For each one, ask: does the destination system need a physical copy of this data, or does it just need to query it? Every pipeline that exists purely for query access is a candidate for virtualization. Replace the pipeline with a federated connection, add a semantic layer for context, and watch your infrastructure costs drop.

Try Dremio Cloud free for 30 days

Batch vs. Streaming: Choose the Right Processing Model

Alex Merced — Wed, 18 Feb 2026 14:00:00 GMT

"We need real-time data." This is one of the most expensive sentences in data engineering — because it's rarely true, and implementing it when it's not needed multiplies complexity, cost, and operational burden.

The question isn't "should we use streaming?" The question is "how fresh does the data actually need to be, and what are we willing to pay for that freshness?"

The Question Isn't "Real-Time or Not" — It's "How Fresh?"

Freshness requirements exist on a spectrum:

Daily (24-hour latency): Fine for financial reporting, historical trend analysis, ML training datasets
Hourly (1-hour latency): Adequate for operational dashboards, inventory tracking, marketing attribution
Near-real-time (1-15 minutes): Sufficient for user activity feeds, recommendation updates, alerting
Real-time (sub-second): Required for fraud detection, stock trading, IoT safety systems

Most "we need real-time" requests are actually "we need hourly" or "we need 5-minute" requests. Clarifying the actual latency requirement before choosing an architecture prevents overengineering.

When Batch Wins

Batch processing is the default choice. Choose it unless you have a specific, justified reason to stream.

Simpler failure recovery. A batch job fails at 3 AM. You fix the bug, rerun the job, and it reprocesses the same bounded dataset. Recovery is predictable and testable.

Easier testing. Given input dataset X, the output should be Y. You can version test datasets, run them locally, and assert exact outputs. Streaming test scenarios require simulating time, ordering, and late-arriving events — dramatically harder.

Lower operational cost. Batch jobs run on schedule, consume resources during execution, and release them when done. Streaming jobs run continuously, consuming resources 24/7 even during low-volume periods.

Better tooling maturity. SQL-based transformations, orchestrators with DAG visualization, version-controlled dbt models — the batch ecosystem is deeper and more mature for most data warehouse workloads.

When to use: Daily/hourly analytics, data warehouse loading, ML training data, compliance reporting, historical backfills.

When Streaming Wins

Streaming processing is the right choice when latency is measured in seconds and the cost of stale data is high.

Fraud detection. You can't batch-process credit card transactions once an hour. By the time you detect a fraudulent pattern, thousands of dollars are already gone. Fraud detection needs event-by-event evaluation in real time.

IoT and safety systems. A temperature sensor in a chemical plant detecting an abnormal reading can't wait for the next hourly batch. Alerting must happen in seconds.

Real-time personalization. Showing a user recommendations based on what they did 30 seconds ago requires streaming user events through a recommendation engine.

Operational systems. Inventory management, ride-sharing pricing, and live logistics tracking all need sub-minute data freshness to function correctly.

When to use: Event-driven business logic, sub-second alerting, real-time user-facing features.

The Micro-Batch Middle Ground

Micro-batch processing runs batch jobs at very short intervals — every 1, 5, or 15 minutes. It captures most of the value of streaming with the simplicity of batch.

Same tools, shorter intervals. Your existing batch infrastructure (SQL transformations, orchestrators, testing frameworks) works unchanged. You just schedule runs more frequently.

Most use cases are satisfied. An operational dashboard refreshing every 5 minutes feels "real-time" to most business users. Marketing attribution updating every 15 minutes is fresh enough for campaign optimization.

Significantly lower complexity. No stream processing framework to learn. No state management. No watermark configuration. No event ordering challenges.

The tradeoff: Micro-batch cannot achieve sub-second latency. If you genuinely need event-by-event processing under one second, you need a streaming framework.

A Decision Framework

Before choosing between batch, micro-batch, and streaming, answer these questions:

Question	Batch	Micro-batch	Streaming
Required latency	Hours	Minutes	Seconds
Cost of stale data	Low	Medium	High
Team streaming expertise	Not needed	Not needed	Required
Operational budget	Lowest	Low	Highest
Recovery complexity	Simple rerun	Simple rerun	Complex

Start with batch. If stakeholders say "we need real-time," ask "what's the cost of a 15-minute delay?" If the answer is "that's fine," micro-batch gives you near-real-time at batch-level complexity.

Upgrade to streaming only when justified. Sub-second latency requirements, event-driven business logic, and high-volume event processing are legitimate streaming use cases. "I want the dashboard to update faster" is usually not.

What to Do Next

List every pipeline in your platform and categorize it by actual (not requested) latency requirement. You'll likely find that 80% or more of your workloads are well-served by batch or micro-batch. Focus streaming investment on the 20% that genuinely needs it.

Try Dremio Cloud free for 30 days

Slowly Changing Dimensions: Types 1-3 with Examples

Alex Merced — Wed, 18 Feb 2026 14:00:00 GMT

Dimensions change. A customer moves cities. A product gets reclassified. An employee changes departments. How your data model handles these changes determines whether your historical reports are accurate or misleading.

Slowly Changing Dimensions (SCDs) are design patterns for managing dimension attribute changes over time. The three most common types — overwrite, track history, and track one change — each make a different tradeoff between simplicity and historical accuracy.

Why Dimensions Change

Dimension tables store descriptive attributes: customer addresses, product categories, employee titles. These attributes don't stay constant. A customer who was in "New York" last quarter is now in "Chicago." A product that was in "Accessories" is now in "Electronics."

If your fact table recorded sales tied to that customer, do last quarter's reports show "New York" (where the customer was at the time of the sale) or "Chicago" (where the customer is now)? The answer depends on your SCD type.

Type 1: Overwrite the Old Value

Type 1 updates the dimension row in place. The old value is gone.

UPDATE dim_customers
SET city = 'Chicago'
WHERE customer_id = 1042;

After this update, every historical fact associated with customer 1042 now appears under "Chicago" — including sales that happened when the customer was in New York.

When to use Type 1:

Correcting errors (fixing a misspelled name)
When historical accuracy for that attribute doesn't matter
When the attribute rarely changes

Tradeoff: No history. If someone asks "How much revenue came from New York customers last quarter?" they get the wrong answer because customer 1042 is now labeled Chicago.

Type 2: Track Full History

Type 2 inserts a new row for each change. The original row is marked as expired, and the new row becomes the current version.

-- Original row (now expired)
-- customer_key: 1042, city: New York, effective_date: 2023-01-15, expiry_date: 2025-03-01, is_current: FALSE

-- New row
INSERT INTO dim_customers (customer_key, customer_id, city, effective_date, expiry_date, is_current)
VALUES (5001, 1042, 'Chicago', '2025-03-01', '9999-12-31', TRUE);

Now each fact row references a specific version of the customer dimension. Sales from Q1 2024 reference customer_key 1042 (New York). Sales from Q2 2025 reference customer_key 5001 (Chicago). Historical reports are accurate.

When to use Type 2:

When historical accuracy matters (most analytics use cases)
When you need to analyze trends by attribute value over time
When regulatory or audit requirements demand change tracking

Tradeoff: The dimension table grows. A customer who changes city three times has three rows. Queries must filter on is_current = TRUE for current-state analysis, or join on date ranges for point-in-time analysis. This adds complexity to every query.

Surrogate keys are essential. The natural business key (customer_id = 1042) appears in multiple rows. A surrogate key (customer_key, auto-incremented) uniquely identifies each version. Fact tables reference the surrogate key, not the natural key.

Type 3: Track One Change

Type 3 adds a column for the previous value instead of adding a row.

ALTER TABLE dim_customers ADD COLUMN previous_city VARCHAR(100);

UPDATE dim_customers
SET previous_city = city, city = 'Chicago'
WHERE customer_id = 1042;

The table now has both city = 'Chicago' and previous_city = 'New York'.

When to use Type 3:

When you need quick access to both the current and immediately prior value
When only one level of history matters
When the dimension changes infrequently

Tradeoff: You only track one change deep. If the customer moves again, the previous value is overwritten. Type 3 is rarely used in practice because most use cases require either no history (Type 1) or full history (Type 2).

Choosing the Right Type

Factor	Type 1 (Overwrite)	Type 2 (New Row)	Type 3 (New Column)
History preserved	No	Full	One level
Dimension growth	No growth	Grows over time	No growth
Query complexity	Simple	Moderate (date filtering)	Simple
Best for	Error corrections	Trend analysis	Before/after comparison
Storage impact	None	Moderate	Minimal
Implementation effort	Low	High	Low

Most analytics organizations use Type 2 as the default and Type 1 for error corrections. Type 3 is a niche choice for specific before/after reporting needs.

In a lakehouse environment, Iceberg's time-travel feature provides an implicit form of historical tracking at the table level. You can query any past snapshot of a table:

SELECT * FROM dim_customers FOR SYSTEM_TIME AS OF '2024-06-15T00:00:00';

This doesn't replace SCD Type 2 (which tracks attribute-level changes with effective dates), but it provides a safety net for point-in-time analysis.

Platforms like Dremio support both approaches. SQL views can present a current-state view (filtering WHERE is_current = TRUE) or an as-of view (joining on effective dates). Wikis document which SCD type each dimension uses, giving AI agents and analysts the context they need to write correct queries.

What to Do Next

Audit your dimension tables. For each one, decide: Does historical accuracy matter for this attribute? If yes, implement Type 2. If the attribute changes rarely and history doesn't matter, Type 1 is sufficient. Document your choice — when the next engineer encounters the dimension, they need to know whether they're looking at current state or historical versions.

Try Dremio Cloud free for 30 days

The Role of the Semantic Layer in Data Governance

Alex Merced — Wed, 18 Feb 2026 14:00:00 GMT

Most organizations have a data governance policy. It lives in a Confluence page. It defines who owns what data, what terms mean, and who should have access. And almost nobody follows it, because it's not enforced where queries actually run.

A semantic layer changes that. It moves governance from a document into the query path, where every rule is applied automatically, for every user, through every tool.

Governance on Paper vs. Governance in Practice

Data governance fails when it depends on people doing the right thing manually. A policy says "Revenue means completed orders minus refunds." An analyst writes a slightly different formula. A dashboard uses the wrong table. An AI agent invents its own definition. The governance policy exists. Nobody follows it. And the organization makes decisions on inconsistent data.

The root cause isn't that people are careless. It's that governance is separated from the systems people actually use to query data. Enforcement happens in a side channel — documentation, review processes, audit logs — not in the query itself.

Centralized Definitions Eliminate Conflicting Metrics

A semantic layer solves the definition problem by making the governance policy code.

Revenue isn't a paragraph in a wiki. It's a SQL view:

CREATE VIEW business.revenue AS
SELECT
    OrderDate,
    Region,
    SUM(OrderTotal) AS Revenue
FROM silver.orders_enriched
WHERE Status = 'completed' AND Refunded = FALSE
GROUP BY OrderDate, Region;

Every dashboard, notebook, and AI agent that needs Revenue queries this view. There's no alternative formula to use. The semantic layer IS the governance for this metric.

When the definition changes (say, a new refund category is added), the view is updated once, and every consumer gets the new logic automatically. No rollout. No migration. No "did everyone update their dashboard?"

Access Policies Enforced at Query Time

The second governance gap: access control. Most organizations enforce security at the BI tool level. Tableau restricts who sees which dashboard. Power BI applies row-level filters. But if someone opens a SQL client and queries the underlying table directly, those filters don't apply.

A semantic layer enforces policies at a lower level. When access control exists in the semantic layer, it applies to every query path:

Query Path	BI-Level Security	Semantic Layer Security
Dashboard	Enforced	Enforced
SQL notebook	Not enforced	Enforced
AI agent	Not enforced	Enforced
API/programmatic access	Not enforced	Enforced

Dremio implements this through Fine-Grained Access Control (FGAC): policies defined as UDFs that filter rows and mask columns based on the querying user's role. These policies are applied at the virtual dataset (view) level. A regional manager queries business.revenue and sees only their region. A data engineer sees all regions. Same view, same SQL, different results based on identity.

This approach eliminates the "security gap" that appears when users bypass BI tools. Every route to the data flows through the semantic layer. Every route inherits the policies.

Lineage and Accountability Through Views

The layered view architecture (Bronze → Silver → Gold) that a semantic layer uses is inherently traceable. Every Gold metric traces back to its Silver business logic, which traces back to the Bronze source mapping, which traces back to raw data.

This traceability matters for compliance. When an auditor asks "Where does your Revenue number come from?", you don't search through dashboards and notebooks. You follow the view chain:

gold.monthly_revenue_by_region → references silver.orders_enriched
silver.orders_enriched → joins bronze.orders_raw with bronze.customers_raw
bronze.orders_raw → maps to production.public.orders in PostgreSQL

Every step is documented. Every transformation is visible. The lineage isn't reconstructed after the fact — it's structural.

Documentation as a Governance Tool

Governance is also about discoverability. Can someone find the right dataset without messaging five people? Can they tell whether a view is production-ready or experimental?

Two mechanisms handle this in a semantic layer:

Wikis attach human-readable (and AI-readable) descriptions to tables, columns, and views. They explain what data represents, where it comes from, and any caveats. A column named cltv gets a description: "Customer Lifetime Value, calculated as total revenue from first purchase to current date, excluding refunds."

Labels categorize data for governance workflows. A label like "PII" triggers automatic column masking. A label like "Certified" indicates the view has been reviewed and approved for production use. A label like "Deprecated" warns consumers to migrate to the replacement.

For organizations with thousands of datasets, manual documentation is impractical. Dremio's generative AI auto-generates Wiki descriptions by sampling table data and suggests Labels based on column content. This bootstraps documentation to 70% coverage automatically. The data team fills in what the AI misses.

Certification and Change Management

Not all views are equal. A semantic layer should distinguish between views that are experimental, under review, and production-ready.

A practical certification workflow:

Draft: New view created by an analyst. Not yet reviewed. Labeled "Draft."
Reviewed: View reviewed by the data team. Business logic validated. Documentation complete.
Certified: View approved for production use. Labeled "Certified." Available in production dashboards and to AI agents.

Each Certified view should have a documented owner — the person accountable for its accuracy and freshness. When business requirements change, the owner updates the view and documentation together. Changes are reviewed before the "Certified" label is reapplied.

This workflow doesn't require advanced tooling. Labels, Wikis, and a team agreement on the process are sufficient. What matters is that governance is visible inside the semantic layer, not tracked in a separate system.

What to Do Next

Audit your top 10 business metrics. For each one, ask three questions: Is the formula defined in one place? Is access control enforced at the query level (not just the BI tool)? Can you trace the number back to its raw source in under 60 seconds? Every "no" is a governance gap that a semantic layer closes.

Try Dremio Cloud free for 30 days

Schema Evolution Without Breaking Consumers

Alex Merced — Wed, 18 Feb 2026 13:00:00 GMT

A source team renames a column from user_id to customer_id. Twelve hours later, five dashboards show blank values, two ML pipelines fail, and the data engineering team spends the morning tracing a problem that could have been prevented with one rule: treat your schema like an API.

Schema evolution is the practice of changing data structures without breaking the systems that depend on them. Get it right, and your data platform stays flexible. Get it wrong, and every schema change becomes an emergency.

Your Schema Is an API

When an application team changes a REST API endpoint, they version it. They deprecate the old version. They give consumers time to migrate. They don't silently rename fields and hope nobody notices.

Data schemas deserve the same discipline. Your columns are fields. Your tables are endpoints. Your downstream consumers — dashboards, ML pipelines, reports, other pipelines — are API clients. When you change the schema, you change the contract.

The difference: API changes are usually intentional and reviewed. Schema changes often happen accidentally — a source system updates its export format, an engineer renames a column for readability, a new data type is introduced. Without guardrails, these changes propagate downstream silently.

Safe vs. Breaking Changes

Not all schema changes carry the same risk:

Backward-compatible (safe) changes:

Adding a new optional column with a default value
Widening a data type (INT to BIGINT, FLOAT to DOUBLE)
Adding documentation or metadata to columns
Reordering columns (if consumers reference by name, not position)

Breaking changes:

Removing a column that consumers reference
Renaming a column without maintaining the old name
Narrowing a data type (BIGINT to INT — values may overflow)
Changing the semantic meaning of a column (e.g., revenue from gross to net)
Changing nullability (nullable to non-nullable breaks inserts with nulls)

The rule: backward-compatible changes can be deployed without coordination. Breaking changes require a migration plan.

The Additive-Only Pattern

The simplest schema evolution strategy: never remove or rename columns. Only add new ones.

When a column needs to be replaced:

Add the new column alongside the old one
Update producers to populate both columns
Migrate consumers to the new column one at a time
Once all consumers have migrated, mark the old column as deprecated
Remove the old column only after a deprecation period (e.g., 90 days)

This pattern is boring — and that's the point. Boring is reliable. Adding a column never breaks existing queries. Consumers that don't need the new column ignore it. Consumers that do need it can adopt it on their own schedule.

Tradeoff: Table width grows over time. Schemas accumulate deprecated columns. This is an acceptable cost compared to production outages.

Schema Versioning and Migration

For changes that can't be additive (fundamental restructuring, data model migrations), use explicit versioning:

Version in the table name. customers_v1, customers_v2 coexist. Consumers migrate from v1 to v2 at their own pace. A view named customers points to the current version.

Version in metadata. Store a schema version field in each record or partition. Consumers check the version and apply the appropriate parsing logic.

Schema registries. Centralized systems that store and validate schemas. Producers register their schema. Consumers declare their expected schema. The registry checks compatibility and rejects breaking changes.

Schema registries enforce rules automatically:

BACKWARD compatible: new schema can read data written by old schema
FORWARD compatible: old schema can read data written by new schema
FULL compatible: both backward and forward compatible

Contract Enforcement at Pipeline Boundaries

Don't rely on conventions ("we don't rename columns"). Enforce contracts programmatically at pipeline boundaries:

At ingestion. Compare the incoming data schema against the expected schema. If columns are missing, added, or retyped, log the difference and alert. For safe changes, proceed and notify. For breaking changes, halt and quarantine.

At transformation. Validate that every column referenced in SQL or transformation logic exists in the input schema. Catch missing-column errors at validation time, not at runtime.

At serving. Validate that output schemas match the contracts expected by consumers. If a downstream dashboard expects column revenue, verify it exists and has the correct type before the pipeline marks the job as successful.

What to Do Next

Document the schema of your five most critical tables: column names, types, nullability, and a one-line description. That's your version 1 contract. Set up an automated check that compares incoming data against this contract and alerts on any deviation. You'll catch the next breaking change before it breaks anything.

Try Dremio Cloud free for 30 days

Dimensional Modeling: Facts, Dimensions, and Grains

Alex Merced — Wed, 18 Feb 2026 13:00:00 GMT

Dimensional modeling is the most widely used approach for organizing analytics data. Developed by Ralph Kimball, it structures data into two types of tables: facts (what happened) and dimensions (the context around what happened). The technique optimizes for query speed and business readability, not for storage efficiency or transactional integrity.

If your goal is to answer business questions quickly and consistently, dimensional modeling is where you start.

Facts and Dimensions: The Two Building Blocks

Fact tables store measurable events. Each row represents something that happened: a sale, a click, a shipment, a login. Fact tables are narrow (a few foreign keys and numeric measures) and deep (millions or billions of rows).

A typical sales fact table might look like:

CREATE TABLE fact_sales (
    sale_id BIGINT,
    date_key INT,
    customer_key INT,
    product_key INT,
    store_key INT,
    quantity INT,
    unit_price DECIMAL(10,2),
    total_amount DECIMAL(12,2)
)

Dimension tables provide context. They describe the "who, what, where, when, and how" behind each fact. Dimension tables are wide (many descriptive columns) and shallow (thousands to millions of rows).

A customer dimension might include: customer_name, email, signup_date, city, state, country, segment, lifetime_value, acquisition_channel.

Every analysis query joins a fact table to one or more dimension tables. "Revenue by region" joins the sales fact to the geography dimension. "Revenue by product category" joins the sales fact to the product dimension. The fact table provides the number; the dimensions provide the labels.

Declaring the Grain

The grain is the most important decision in dimensional modeling. It declares what one row in your fact table represents.

"One row per order line item" — each product within an order gets its own row
"One row per daily customer session" — each customer's daily activity is aggregated into one row
"One row per monthly account balance" — snapshot taken once per month

Getting the grain right matters because:

Too coarse: You lose detail. If your grain is "one row per order" you can't analyze individual line items.
Too fine: You create an enormous table that's expensive to query. If your grain is "one row per page view" in a high-traffic application, the table grows by billions of rows per month.
Inconsistent: If some rows represent individual items and others represent aggregated totals, every calculation produces wrong results.

Declare the grain first. Then identify which dimensions apply at that grain, and which numeric measures belong in the fact table. This order is not optional — skip it, and the model breaks down.

Designing Fact Tables

Three types of fact tables handle different analytical patterns:

Transaction facts record individual events. One row per sale, one row per click. This is the most common type. It supports the most detailed analysis but produces the largest tables.

Periodic snapshot facts capture the state at regular intervals. One row per account per month. Useful for balance-tracking, inventory levels, and any measure that accumulates over time.

Accumulating snapshot facts track the lifecycle of a process. One row per order, with date columns for each milestone (order_placed, payment_received, shipped, delivered). Useful for analyzing process efficiency and bottleneck identification.

Best practices for fact tables:

Keep facts additive when possible (SUM-able across dimensions)
Avoid storing text in fact tables — that belongs in dimensions
Use surrogate keys (integers) for dimension references, not natural keys
Never mix grains in one fact table

Designing Dimension Tables

Well-designed dimensions follow predictable patterns:

Denormalize. Include all descriptive attributes in one table. Product name, category, subcategory, brand, manufacturer, department — all in dim_products. This eliminates joins and makes queries readable.

Use surrogate keys. Assign an integer key (product_key) that acts as the primary key. Keep the natural business key (product_sku) as a regular attribute. Surrogate keys insulate your model from source system key changes.

Add audit columns. Include effective_date, expiry_date, and is_current flag for tracking changes over time (Slowly Changing Dimensions — covered in a separate article).

Include "junk" dimensions. Low-cardinality flags and indicators (is_promotional, is_online, payment_type) can be combined into a single "junk dimension" instead of cluttering the fact table.

Conformed Dimensions

A conformed dimension is shared across multiple fact tables. The best example is the Date dimension — every fact table references dates, and they should all use the same date dimension to ensure consistent filtering and grouping.

Other conformed dimensions: Customer, Product, Employee, Geography. When Sales and Support both reference the same dim_customers table, you can analyze customer behavior across both domains without reconciling different customer definitions.

Conformed dimensions are the connective tissue of a dimensional model. Without them, each fact table exists in isolation.

Platforms like Dremio support dimensional modeling through virtual datasets. Fact and dimension views live in the Silver layer of a Medallion Architecture. Conformed dimensions are defined once and referenced by multiple fact views. Wikis document what each dimension attribute means, and AI agents use that documentation to generate accurate queries.

What to Do Next

Start your dimensional model with one business process — the one your team queries most. Declare the grain. Identify the dimensions. Build the fact table. Then expand: pick the next business process, reuse the conformed dimensions, and add new ones as needed.

Try Dremio Cloud free for 30 days

Why Your AI Initiatives Fail Without a Semantic Layer

Alex Merced — Wed, 18 Feb 2026 13:00:00 GMT

Your team builds an AI agent. It connects to your data warehouse. A product manager types "What was revenue last quarter?" and gets a number. The number is wrong. Nobody knows it's wrong until Finance runs the same query manually and gets a different result.

This happens constantly. And the problem isn't the AI model. It's the missing layer between the model and your data.

The Promise vs. the Reality

Natural language analytics is the most requested feature in every data platform survey. Business users want to ask questions in plain English and get accurate answers. No SQL. No tickets. No waiting.

The technology exists. Large language models can generate SQL from natural language with impressive accuracy. But accuracy on syntax isn't accuracy on meaning. An LLM can write grammatically correct SQL that returns the wrong answer because it doesn't understand your business definitions.

A semantic layer provides those definitions. Without one, AI analytics is a demonstration that works in a meeting but fails in production.

Five Ways AI Goes Wrong Without a Semantic Layer

Metric Hallucination

Your LLM decides that Revenue = SUM(amount) from the transactions table. But your actual Revenue formula is SUM(order_total) WHERE status = 'completed' AND refunded = FALSE from the orders table. The AI's number is plausible. It's also wrong by 15%. Nobody catches it because it looks reasonable.

The fix: Canonical metric definitions in virtual datasets. The AI references the view, not its own invented formula.

Join Confusion

There are three paths from orders to customers: via customer_id, via billing_address_id, and via shipping_address_id. For revenue analysis, you want customer_id. The LLM picks billing_address_id because it seems logical. The numbers are close enough that the mistake survives review.

The fix: Pre-defined join relationships in the semantic model. The AI follows the approved path.

Column Misinterpretation

A column called date appears in the orders table. Is it the order date, ship date, or invoice date? The LLM guesses order date. It's actually the ship date. Every time-based query is off by 2-5 days.

The fix: Wiki descriptions on every column. The semantic layer tells the AI that date is ShipDate and OrderDate is the field to use for time-based revenue analysis.

Security Bypass

Your BI dashboard applies row-level security so regional managers only see their region's data. The AI agent queries the raw table directly, bypassing the BI layer. A regional manager asks about "their" revenue and sees the entire company's numbers.

The fix: Fine-Grained Access Control enforced at the semantic layer. The AI queries views, not raw tables. Security policies travel with the data regardless of the access path.

Inconsistent Results

The same question asked twice generates different SQL because the LLM's output is probabilistic. Monday's answer: $4.2M. Wednesday's answer: $4.5M. Both are "correct" given the SQL generated. Neither matches Finance's number.

The fix: Deterministic definitions in the semantic layer. The same question always resolves to the same view, the same formula, the same result.

How a Semantic Layer Grounds AI

Each failure maps to a specific semantic layer component:

Failure Mode	Semantic Layer Fix
Metric hallucination	Virtual datasets with canonical formulas
Join confusion	Pre-defined join relationships
Column misinterpretation	Wiki descriptions on every field
Security bypass	Access policies enforced at the view level
Inconsistent results	Deterministic definitions (same question = same SQL)

This is why platforms that take AI analytics seriously embed the semantic layer directly into the query engine. Dremio's approach combines virtual datasets, Wikis, Labels, and Fine-Grained Access Control into a single layer that both humans and AI agents consume. The AI doesn't just generate SQL. It consults the semantic layer to understand what the data means, which formulas to use, and what the querying user is allowed to see.

What AI-Ready Architecture Looks Like

An AI-ready data platform doesn't just connect an LLM to a database. It puts a structured context layer in between:

Semantic layer defines metrics, documents columns, and enforces security
AI agent reads the semantic layer to understand business context
Query engine executes the AI-generated SQL with full optimization (caching, reflections, pushdowns)
Results are returned in business terms through the same interface humans use

Without step 1, the AI is just a SQL autocomplete tool with no business understanding. It writes syntactically valid queries that produce semantically wrong answers. The semantic layer is the difference between a toy demo and a production-grade AI analytics system.

What to Do Next

If your AI analytics initiative is producing unreliable results, don't upgrade the model. Audit the context the model has access to. Can it read your metric definitions? Column descriptions? Security policies? If the answer is no, the fix isn't a better LLM. It's a semantic layer.

Try Dremio Cloud free for 30 days

Idempotent Pipelines: Build Once, Run Safely Forever

Alex Merced — Wed, 18 Feb 2026 12:00:00 GMT

A pipeline runs, processes 100,000 records, and loads them into the target table. Then it fails on a downstream step. The orchestrator retries the entire job. Now the table has 200,000 records — 100,000 of them duplicates. Revenue reports double. Dashboards misfire. Someone spends the next four hours manually deduplicating records and explaining to stakeholders why the numbers were wrong.

This is the cost of not building idempotent pipelines.

What Idempotency Means for Pipelines

An idempotent operation produces the same result no matter how many times you execute it. For data pipelines, that means: running the same job twice — or ten times — leaves the target data in the exact same state as running it once.

This property matters because retries are inevitable. Orchestrators retry failed tasks. Backfill jobs reprocess historical data. Network glitches cause at-least-once delivery. Engineers manually rerun jobs during debugging. Without idempotency, every one of these events risks data corruption.

Idempotency is not about preventing retries. It's about making retries safe.

The Partition Overwrite Pattern

The simplest and most reliable idempotency pattern for batch pipelines: overwrite the entire partition.

Instead of appending rows, your pipeline replaces the complete partition for the time period being processed. For a daily pipeline processing January 15th:

-- Delete existing data for this partition
DELETE FROM target_table WHERE event_date = '2024-01-15';

-- Insert fresh data for this partition
INSERT INTO target_table
SELECT * FROM staging_table WHERE event_date = '2024-01-15';

If the job reruns, it deletes and recreates the same partition — resulting in the same data. Many table formats support INSERT OVERWRITE or REPLACE PARTITION as an atomic operation, which is even safer because it avoids a window where the partition is empty.

When to use it: Daily, hourly, or other time-partitioned batch pipelines. This covers the majority of data warehouse loading patterns.

Limitation: You need a clear partitioning key. For non-time-series data, partition overwrite may not apply.

The Upsert/MERGE Pattern

For data that doesn't partition cleanly — or for change data capture (CDC) workloads — use MERGE (also called upsert):

MERGE INTO target_table t
USING staging_table s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET
  t.status = s.status,
  t.updated_at = s.updated_at
WHEN NOT MATCHED THEN INSERT (order_id, status, updated_at)
VALUES (s.order_id, s.status, s.updated_at);

If the merge runs twice with the same staging data, the result is identical. Existing records update to the same values. New records insert once because they already exist on the second run.

When to use it: CDC pipelines, entity-centric data (customers, products, accounts), and slowly changing dimensions.

Requirement: A reliable business key that uniquely identifies each record. Without one, merges produce inconsistent results.

Event Deduplication for Streaming

Streaming systems typically guarantee at-least-once delivery, which means the same event can be delivered and processed multiple times. Your pipeline needs to handle this.

Process-level deduplication. Maintain a set (in-memory, in a key-value store, or in the target database) of recently processed event IDs. Before processing each event, check if its ID has been seen. Skip duplicates.

Write-level deduplication. Use MERGE or conditional INSERT:

INSERT INTO events (event_id, payload, processed_at)
SELECT event_id, payload, NOW()
FROM incoming_events
WHERE event_id NOT IN (SELECT event_id FROM events);

Windowed deduplication. For high-volume streams, maintain dedup state only for a window (e.g., last 24 hours). Events outside the window are assumed to be unique — a practical tradeoff between memory usage and dedup completeness.

Anti-Patterns That Break Idempotency

Blind INSERT/APPEND. Every retry adds duplicate rows. This is the default behavior in most systems and the most common cause of data inflation.

Auto-incrementing surrogate keys. If your pipeline generates IDs at processing time (not from the source data), duplicates get different IDs and look like distinct records.

Timestamps as dedup keys. Using processed_at as part of the primary key means the same source record processed at different times produces different target records.

"We'll dedup later." Deferring deduplication to a cleanup job means every consumer between the load and the cleanup sees dirty data.

What to Do Next

Identify your five most frequently retried or backfilled pipelines. Check whether they use INSERT or MERGE. If they use INSERT, switch to partition overwrite or MERGE. Run the pipeline twice intentionally and verify the target table has the same row count both times. That's your idempotency test.

Try Dremio Cloud free for 30 days

Data Modeling for the Lakehouse: What Changes

Alex Merced — Wed, 18 Feb 2026 12:00:00 GMT

Traditional data modeling assumed you controlled the database. You defined schemas up front, enforced foreign keys at write time, and optimized with indexes. The lakehouse changes every one of those assumptions.

Data lives in open file formats on object storage. Schemas evolve without rewriting data. Queries run through engines that may not enforce relational constraints. The modeling discipline is the same, but the mechanics are different.

What's Different About a Lakehouse

A lakehouse stores data as files — typically Parquet — on object storage like S3 or Azure Blob. An open table format like Apache Iceberg adds structure: schema definitions, partition metadata, snapshot history, and transactional guarantees.

This architecture gives you more flexibility than a traditional RDBMS, but also more responsibility. There are no foreign key constraints enforced at write time. No triggers. No stored procedures. Referential integrity is your problem to solve in pipelines and views, not something the storage engine handles for you.

The tradeoff is worth it: open formats, engine portability, cheap storage, and the ability to run multiple compute engines (Spark, Dremio, Flink, Trino) against the same data.

Schema-on-Read Changes the Rules

In a traditional warehouse, you define the schema before writing data (schema-on-write). Every row must conform to the schema or the insert fails. This guarantees consistency but makes changes expensive. Adding a column means an ALTER TABLE. Changing a data type might require rewriting the entire table.

In a lakehouse, you can also store data first and apply structure at query time (schema-on-read). Iceberg supports schema evolution natively — add columns, rename columns, widen data types, and reorder fields without rewriting underlying files.

This flexibility changes how you model:

Bronze layer: Accept data as-is from sources. Apply minimal typing. Don't reject records that don't match a rigid schema.
Silver layer: Apply business logic, joins, and type enforcement through SQL views.
Gold layer: Serve consumption-ready datasets with stable, documented schemas.

The model evolves at the view layer, not the storage layer. This makes iteration faster and migration cheaper.

The Medallion Architecture as a Modeling Pattern

The Medallion Architecture (Bronze → Silver → Gold) is the most common data modeling pattern in lakehouse environments. Each layer is a set of SQL views or managed tables:

Bronze (Preparation):

Maps raw source data to typed columns
Renames ambiguous column names
Applies basic data type casting
One view per source table

Silver (Business Logic):

Joins related entities (orders + customers + products)
Applies business rules (revenue = quantity × price WHERE status = 'completed')
Filters invalid or duplicate records
Implements the logical data model

Gold (Application):

Tailored views for specific use cases
Executive dashboards, Sales reports, AI agent context
Minimal transformation — mostly selecting from Silver views
Documented with business-friendly names and descriptions

In Dremio, these layers are implemented as virtual datasets (SQL views) organized in Spaces. Each view is documented with Wikis, tagged with Labels, and governed with Fine-Grained Access Control. The logical model lives in the platform, not in scattered dbt files or tribal knowledge.

Physical Modeling for Iceberg Tables

When you do create physical Iceberg tables (as opposed to views), the modeling considerations differ from traditional RDBMS:

Partitioning matters more than indexing. Iceberg uses partition pruning instead of traditional B-tree indexes. Choose partition columns based on your most common query filters — typically date columns. Iceberg's hidden partitioning means users don't need to know the partition scheme to write efficient queries.

Sort order affects scan performance. Within each partition, Iceberg can sort data by specified columns. Sorting by a frequently filtered column (like customer_id or region) enables min/max pruning that skips irrelevant files.

Compaction replaces vacuum. Small files accumulate from streaming inserts. Regular compaction rewrites many small files into fewer large files, improving scan performance.

Schema evolution is non-destructive. Adding a column to an Iceberg table doesn't rewrite existing files. Old files return null for the new column. This makes the physical model more adaptable than traditional databases.

Challenges to Watch For

No referential integrity enforcement. The lakehouse won't stop you from inserting an order with a customer_id that doesn't exist in the customers table. Build data quality checks in your pipelines.

Schema drift across sources. When sources change their schemas unexpectedly, your Bronze layer must handle it. Design Bronze views to be tolerant of new or missing columns.

Over-reliance on views. Views are powerful, but deeply nested views (View D reads from View C reads from View B reads from View A) create performance and debugging challenges. Keep the chain to three levels when possible.

What to Do Next

If you're moving from a traditional warehouse to a lakehouse, start by recreating your most-used tables as Iceberg tables and your most-used transformations as SQL views. Organize those views into Bronze, Silver, and Gold layers. Measure whether query performance meets your SLAs — and if it doesn't, add Reflections to optimize the heavy queries without changing the logical model.

Try Dremio Cloud free for 30 days

Semantic Layer vs. Data Catalog: Complementary, Not Competing

Alex Merced — Wed, 18 Feb 2026 12:00:00 GMT

"We already have a data catalog, so we don't need a semantic layer." This is one of the most common misconceptions in modern data architecture. Catalogs and semantic layers both deal with metadata. They both improve data accessibility. But they solve fundamentally different problems.

Swapping one for the other leaves a critical gap in your stack.

What a Data Catalog Does

A data catalog is a searchable inventory of your organization's data assets. Think of it as a library card system for data. It tells you what data exists, where it lives, who owns it, and how it flows through your systems.

Key functions:

Discovery: Find tables, views, files, and dashboards by searching keywords, tags, or owners
Lineage: Trace how data moves from source to destination, including every transformation along the way
Governance metadata: Track data quality scores, classification (PII, confidential), and compliance status
Documentation: Store descriptions of assets, often crowd-sourced from data producers and consumers

A data catalog is fundamentally a passive system. You search it, browse it, and read from it. It doesn't change how queries execute or how metrics are calculated. It organizes information about data.

What a Semantic Layer Does

A semantic layer defines what data means and how to use it correctly. It's an active system that sits between your raw data and the tools querying it.

Key functions:

Metric definitions: Revenue, Churn Rate, Active Users — calculated one way, everywhere
Query translation: Converts business questions into optimized SQL
Access enforcement: Row-level security and column masking applied at query time
Documentation: Wikis and labels attached to views and columns

A semantic layer actively participates in every query. When a user asks "What was revenue by region?", the semantic layer translates "revenue" into the correct SQL formula, joins the right tables, applies security filters, and returns the result.

Side-by-Side Comparison

Dimension	Data Catalog	Semantic Layer
Primary question answered	"What data do we have?"	"What does this data mean?"
System behavior	Passive (search & browse)	Active (query translation)
Scope	All metadata across assets	Business definitions, metrics, security
Lineage	Tracks data flow	Defines calculation logic
Query execution	Does not execute queries	Translates and optimizes queries
Access control	Documents policies	Enforces policies at query time

The catalog tells you a table called orders exists in the production schema. The semantic layer tells you that "Revenue" means SUM(orders.total) WHERE status = 'completed', joins it to customers on customer_id, and filters results based on the querying user's role.

Why You Need Both

A catalog without a semantic layer: Users find data but don't know how to use it correctly. They discover the orders table but write their own revenue formula, which differs from the formula Finance uses. Data is discoverable but inconsistently interpreted.

A semantic layer without a catalog: Users get accurate, governed queries for the datasets the semantic layer covers. But they can't discover datasets outside the layer. New data sources, experimental tables, and raw files remain invisible until someone manually adds views.

The best architectures integrate both. The catalog handles discovery and lineage across everything. The semantic layer handles meaning, calculation, and governance for the business-critical datasets that drive decisions.

What Integration Looks Like

An integrated system gives you a single interface where data discovery and business context exist side by side. You search the catalog to find a dataset. You see its semantic layer definition — the metric formulas, documentation, labels, and access policies — alongside the catalog metadata (lineage, quality, ownership).

Dremio achieves this with its Open Catalog (built on Apache Polaris, the open-source Iceberg REST catalog standard) combined with its semantic layer features:

Open Catalog provides the inventory: tables, views, sources, and their lineage
Virtual datasets (SQL views) define business logic and metric calculations
Wikis document what each dataset and column means
Labels tag data for governance and discoverability (PII, Finance, Certified)
FGAC enforces row/column security at query time

AI agents benefit from this integration directly. They use the catalog to navigate available datasets (what tables exist in the "Sales" space?) and the semantic layer to generate accurate queries (what does "Revenue" mean, and who can see which rows?). Remove either piece, and the AI is either blind to available data or confidently generating wrong SQL.

What to Do Next

Open your current data catalog and pick a business-critical table. Can you see how its key metric is calculated? Who can access which rows? What the column names mean in business terms? If the catalog only shows you that the table exists, you've identified the gap a semantic layer fills.

Try Dremio Cloud free for 30 days

Data Quality Is a Pipeline Problem, Not a Dashboard Problem

Alex Merced — Wed, 18 Feb 2026 11:00:00 GMT

When an analyst finds null values in a revenue column, the typical response is to add a calculated field in the BI tool: IF revenue IS NULL THEN 0. That "fix" doesn't fix anything. It masks a problem at the source — and every downstream consumer has to independently discover and patch the same issue.

Data quality is a pipeline problem. It should be enforced where data enters your system, not where it exits as a chart.

The Dashboard Isn't Where Quality Gets Fixed

Quality problems that surface in dashboards have already propagated through every layer of your stack: raw tables, transformed models, aggregations, caches, and API endpoints. By the time an analyst spots a zero-revenue row, the bad record has been used to train ML models, trigger automated alerts, and populate executive reports.

Fixing quality at the point of consumption is reactive, fragmented, and unrepeatable. Every team applies different patches. Every new consumer rediscovers the same problems.

Fixing quality at the point of ingestion is proactive, centralized, and consistent. Every downstream consumer benefits from the same validated data.

Six Dimensions of Data Quality

Not all quality problems are the same. Categorizing them helps you build targeted checks:

Completeness. Are required fields populated? A customer record missing an email address might be acceptable. A transaction record missing an amount is not.

Accuracy. Do values reflect reality? An age of 250 is syntactically valid but factually wrong. Accuracy checks require domain knowledge and range validation.

Consistency. Do the same facts agree across sources? If your CRM says a customer is in Texas and your billing system says California, you have a consistency problem.

Timeliness. Did the data arrive when expected? A daily feed that arrives 6 hours late might still be correct — but any dashboards refreshed before it arrived showed stale numbers.

Uniqueness. Are there duplicate records? Double-counted revenue is worse than no revenue. Deduplication on business keys (order ID, event ID) is essential.

Validity. Do values conform to expected formats and ranges? Dates in the future, negative quantities, email addresses without @ signs — structural validation catches these before they corrupt downstream logic.

Enforce Quality Inside the Pipeline

Add a validation stage between ingestion and transformation. This stage checks every record against defined quality rules and routes failures to a quarantine table.

Schema validation. Check column names, data types, and required vs. optional fields. If the source adds or removes a column, catch it here — not when a transformation SQL query fails.

Range and format checks. Ensure numeric values fall within expected ranges (0 ≤ price ≤ 1,000,000). Validate date formats, email patterns, and enum values against allowed lists.

Referential checks. Verify that foreign key values exist in their reference tables. An order referencing a non-existent customer ID means either the order is invalid or the customer pipeline is behind.

Volume checks. Compare the row count of the incoming batch against historical baselines. A daily feed that usually delivers 50,000 rows but arrives with 500 rows should trigger an alert, not proceed silently.

Freshness checks. Validate that event timestamps fall within the expected window. A batch of events all timestamped from three days ago may indicate a delayed replay, not current data.

Quarantine, Don't Drop

When a record fails validation, don't drop it. Route it to a quarantine table with metadata: which check failed, when, and the original record content.

Dropping bad records silently creates invisible data loss. Your row counts won't match, your aggregations will undercount, and no one will know why.

Quarantined records give you:

Visibility. You know how many records failed and why.
Recovery. When the quality rule was too strict (false positive), you can reprocess quarantined records.
Root cause analysis. Patterns in quarantine (e.g., all failures from one source) help you fix the actual problem upstream.
Accountability. You can report quality rates per source, per pipeline, per day.

Track Quality Like You Track Uptime

Pipeline monitoring typically covers: did the job run? Did it succeed? How long did it take? Quality monitoring adds: how many records passed validation? What percentage failed? Which checks triggered the most failures?

Build quality metrics into your monitoring dashboards:

Pass/fail ratio per pipeline, per day
Failure breakdown by quality dimension (completeness, accuracy, etc.)
Trend lines to catch gradual degradation before it becomes critical
SLA tracking for freshness and completeness targets

Alert on quality regressions the same way you alert on pipeline failures. A pipeline that runs successfully but produces 30% invalid records is worse than one that fails outright — because it's silently wrong.

What to Do Next

Audit your most important pipeline. Add a validation stage with checks for completeness, uniqueness, and volume. Route failures to a quarantine table. Within a week, you'll know more about your data quality than any dashboard could tell you.

Try Dremio Cloud free for 30 days

Star Schema vs. Snowflake Schema: When to Use Each

Alex Merced — Wed, 18 Feb 2026 11:00:00 GMT

Both star schemas and snowflake schemas are dimensional models. They both organize data into fact tables (measurable events) and dimension tables (context about those events). The difference is how they structure the dimensions.

That structural difference affects query performance, storage efficiency, SQL complexity, and how easily BI tools and AI agents can interpret your data. Here's how to choose.

The Two Patterns of Dimensional Modeling

Dimensional modeling separates data into two types:

Fact tables store measurable events — a sale, a page view, a shipment, a login. Each row represents one event. Columns include numeric measures (revenue, quantity, duration) and foreign keys pointing to dimension tables.

Dimension tables provide context for facts — who (customer), what (product), when (date), where (location), how (channel). Dimensions describe the "business words" people use to filter, group, and label their analysis.

Star and snowflake schemas differ in how they organize those dimension tables.

Star Schema: Denormalized Dimensions

In a star schema, each dimension is a single, denormalized table. All attributes for a dimension live in one place.

A product dimension contains the product name, category, subcategory, department, and brand — all in one table. This means some values repeat. Every product in the "Electronics" category stores the string "Electronics" in its row.

Advantages:

Fewer joins per query. A typical star schema query joins the fact table to 3-5 dimension tables. That's it.
Simpler SQL. Analysts write shorter, more readable queries.
Faster query performance. Fewer joins means less work for the query engine.
Better BI tool compatibility. Most BI tools expect star schemas and generate optimal SQL against them.

Tradeoff: Data redundancy in dimensions. If the "Electronics" department changes its name, you update it in every row that references it.

Snowflake Schema: Normalized Dimensions

In a snowflake schema, dimensions are normalized into sub-tables. Instead of one product dimension, you have separate tables for Product, Category, Subcategory, and Department, linked by foreign keys.

Advantages:

Less storage redundancy. Each value stored once. "Electronics" appears in one row of the Department table.
Single source of truth per attribute. Rename a department in one row instead of thousands.
Aligns with OLTP normalization practices. Familiar to engineers coming from transactional database backgrounds.

Tradeoff: More joins per query. A query that would join 4 tables in a star schema might join 8-12 tables in a snowflake schema. SQL gets longer, more complex, and harder for analysts to write without help.

Side-by-Side Comparison

Aspect	Star Schema	Snowflake Schema
Dimension structure	Denormalized (flat)	Normalized (branching)
Tables per query	Fewer (4-6 typical)	More (8-12 typical)
Query performance	Faster	Slower (more joins)
SQL complexity	Simpler	More complex
Storage efficiency	Lower (some redundancy)	Higher
BI tool compatibility	Better	Harder
ETL/pipeline complexity	Simpler loads	More complex loads
Self-service friendliness	High	Low
Update granularity	Update many rows	Update one row

When to Choose Which

Choose a star schema when:

Your primary workload is analytics and reporting
Business users run ad-hoc queries or use BI tools
Query performance matters more than storage costs
You want AI agents to generate accurate SQL (fewer joins = fewer mistakes)
Your dimensions are small enough that redundancy is negligible

Choose a snowflake schema when:

Dimensions are very large and redundancy has real storage costs
Regulatory requirements demand a single canonical source per attribute
Only ETL engineers (not analysts) write queries against the model
You need strict referential integrity across dimension hierarchies

Why Star Schema Usually Wins

Three changes in modern data platforms have tilted the balance toward star schemas:

Storage is cheap. Object storage costs a fraction of a cent per gigabyte per month. The storage savings from normalizing dimensions rarely justify the query complexity cost.

Columnar formats compress redundancy well. Parquet and ORC store data in columns. Repeated values like "Electronics" compress to nearly nothing. The physical storage overhead of a denormalized dimension is much smaller than it appears in row-oriented thinking.

AI and self-service need simplicity. When an AI agent generates SQL against your data model, fewer tables and fewer joins reduce the chance of hallucinated join paths. When a business analyst builds a report, fewer joins reduce the chance of wrong results.

Platforms like Dremio make this choice even easier. Virtual datasets let you model star schemas as SQL views without physically copying or denormalizing data. Reflections automatically optimize query performance in the background. You get the simplicity of a star schema with optimized physical performance, regardless of how the underlying data is stored.

What to Do Next

Take your most-used fact table. Count the joins required to build a complete report. If you're joining more than five dimension tables, or if dimension tables themselves require sub-joins, consider flattening your dimensions into a star schema. Measure the query performance difference. In most cases, the improvement is significant and the storage increase is negligible.

Try Dremio Cloud free for 30 days

Semantic Layer vs. Metrics Layer: What's the Difference?

Alex Merced — Wed, 18 Feb 2026 11:00:00 GMT

Both terms appear in every modern data architecture diagram. They're used interchangeably in conference talks, Slack threads, and vendor marketing. And almost nobody defines them precisely.

Here's the difference, why it matters, and what it means for how you build your data platform.

What a Metrics Layer Does

A metrics layer has one job: define how business metrics are calculated and make those definitions available to every tool in your stack.

Take Revenue. Without a metrics layer, the formula lives in a dashboard filter, a dbt model, a Python notebook, and three different analysts' heads. With a metrics layer, the formula is defined once:

Revenue = SUM(order_total) WHERE status = 'completed' AND refunded = FALSE

Every dashboard, API endpoint, and AI agent that needs "Revenue" pulls from this single definition. Change the formula in one place, and it updates everywhere.

Metrics layers are typically code-defined. dbt's semantic layer uses YAML specifications. Cube.js uses JavaScript schemas. The metric definition includes the calculation, the time dimension, the allowed filters, and the grain.

This is valuable. But it's incomplete.

What a Semantic Layer Does

A semantic layer does everything a metrics layer does, plus more. It covers the full abstraction between raw data and the people (and machines) querying it.

Capability	Metrics Layer	Semantic Layer
Metric definitions (KPI calculations)	Yes	Yes
Documentation (table/column descriptions)	Sometimes	Yes
Labels and tags (governance, discoverability)	No	Yes
Join relationships (pre-defined paths)	Limited	Yes
Access policies (row/column security)	No	Yes
Query optimization (caching, pre-aggregation)	No	Often

A metrics layer tells you how to calculate a number. A semantic layer tells you what the data means, how to calculate it, who can see it, how to join it, and where it came from.

The Relationship: Subset, Not Alternative

A metrics layer is a component of a semantic layer. Not a replacement.

Think of it like a spreadsheet. The metrics layer is the formulas: revenue calculations, growth rates, ratios. The semantic layer is the entire workbook: formulas, column headers, sheet labels, formatting, and sharing permissions. You can't have a useful workbook with just formulas. And you can't have a complete semantic layer without metric definitions.

The confusion arose because different vendors built different pieces first. dbt built the metrics layer and called it a "semantic layer." BI tools like Looker built semantic models (LookML) focused on relationships and query patterns. Platforms like Dremio built a full semantic layer that includes views, documentation, governance, and AI context in one integrated system.

Why the Distinction Matters

If you build a metrics layer but skip the rest of the semantic layer, you leave three gaps:

No documentation means no AI accuracy. When an AI agent generates SQL, it needs more than metric formulas. It needs to know what each column represents, which tables to join, and what filters are valid. Metric definitions alone don't provide that. Wikis, labels, and column descriptions do. Without them, AI agents hallucinate joins and misinterpret fields.

No security means enforcement happens ad hoc. A metrics layer doesn't include row-level security or column masking. Those policies get applied separately in each BI tool, each notebook, each API. One missed policy, and sensitive data leaks to the wrong role.

No join paths means redundant work. If the metrics layer defines "Revenue" but doesn't define how to connect the Orders table to the Customers table, every consumer figures out the join independently. Some get it right. Some don't. You get conflicting results from a formula that was supposed to be centralized.

What This Looks Like in Practice

A platform with a full semantic layer, like Dremio, provides:

Virtual datasets (SQL views) that define business logic across federated sources
Wikis that document tables and columns in human- and AI-readable format
Labels that tag data for governance (PII, Finance, Certified)
Fine-Grained Access Control that enforces row/column security at the view level
Reflections that automatically optimize performance for the most-queried views
AI-generated metadata that auto-populates descriptions and label suggestions

Compare that to a standalone metrics layer, which gives you metric definitions and (sometimes) basic documentation. The metrics layer is the engine. The semantic layer is the complete vehicle.

What to Do Next

If you already have a metrics layer, audit what's missing. Do your metric definitions include documentation? Labels? Security policies? Join paths? If not, you have a piece of the semantic layer, not the whole thing.

Completing the picture means either extending your metrics layer with those capabilities, or adopting a platform that provides them natively.

Try Dremio Cloud free for 30 days

How to Design Reliable Data Pipelines

Alex Merced — Wed, 18 Feb 2026 10:00:00 GMT

Most pipeline failures aren't caused by bad code. They're caused by no architecture. A script that reads from an API, transforms JSON, and writes to a database works fine on day one. On day ninety it fails at 3 AM because the API changed its response format, and the only way to recover is to rerun the entire pipeline from scratch — hoping that reprocessing three months of data doesn't create duplicates.

Reliable pipelines are designed, not debugged into existence.

Reliability Is a Design Property, Not a Bug-Fix

You don't make a pipeline reliable by adding try-catch blocks after it breaks. You make it reliable by building reliability into the architecture from the start. That means:

Resumability. After a failure, you restart from where it stopped, not from the beginning.
Idempotency. Running the same job twice produces the same result.
Observability. You know what the pipeline processed, how long it took, and where it is right now.
Isolation. One failing stage doesn't cascade into unrelated stages.

These properties don't come from choosing the right framework. They come from how you structure the pipeline.

The Four Architecture Layers

Every well-designed pipeline has four distinct layers, even if they run in the same job:

Ingestion. Pull raw data from sources and land it unchanged. Don't transform here. Don't filter. Don't join. Store the raw data exactly as it arrived, with metadata (timestamp, source, batch ID). This gives you a replayable audit trail.

Staging. Validate the raw data. Check for schema compliance, null values in required fields, duplicate records, and data type mismatches. Records that fail validation go to a quarantine table or dead-letter queue — they don't silently disappear.

Transformation. Apply business logic: joins, aggregations, calculations, enrichments. This is where raw events become metrics, where customer records merge across sources, where timestamps convert to business periods. Keep business logic in one layer, not spread across ingestion and loading scripts.

Serving. Organize the transformed data for consumers. Analysts need star schemas. ML models need feature tables. APIs need denormalized lookups. The serving layer shapes data for its audience without changing the underlying transformation logic.

Build a DAG, Not a Script

A script runs steps in order: step 1, step 2, step 3. If step 2 fails, you rerun from step 1. If step 3 needs a new input, you rewrite the script.

A directed acyclic graph (DAG) models dependencies explicitly. Step 3 depends on step 2 and step 4. Step 2 and step 4 can run in parallel. If step 2 fails, you rerun step 2 — not steps 1, 4, or 3.

DAG-based thinking gives you:

Parallelism. Independent stages run concurrently, cutting wall-clock time.
Targeted retries. Failed stages retry alone, not the entire workflow.
Clear dependencies. You can see exactly what feeds into a given output.
Incremental development. Add new stages without touching existing ones.

Even if your orchestrator doesn't enforce DAGs, design your pipeline as one. Document which stages depend on which outputs. Make each stage read from a defined input location and write to a defined output location.

Dependency Management

Implicit dependencies are the most common source of pipeline fragility. "This pipeline assumes table X exists because another pipeline created it" is an implicit dependency. When the other pipeline is delayed, skipped, or renamed, your pipeline breaks.

Make dependencies explicit:

Declare data dependencies. If stage B reads the output of stage A, model that relationship in your orchestration. Don't rely on timing ("A usually finishes by 6 AM").
Use sensors or triggers. Wait for data to arrive before starting a stage. Check for a file, a partition, or a row count — don't check the clock.
Version your interfaces. When a producer changes its output schema, consumers should detect the change before they process stale or malformed data.
Document ownership. Every dataset should have an owner. When you depend on someone else's table, you should know who to contact when it changes.

Failure Handling Patterns

Retry with backoff. Most transient failures (network timeouts, API throttling, lock contention) resolve themselves. Retry 3-5 times with exponential backoff (e.g., 1s, 5s, 25s) before marking a stage as failed.

Dead-letter queues. Records that cannot be processed (corrupt payloads, unexpected schemas, values out of range) go to a quarantine area. Log why they failed. Review them periodically. Don't drop them silently.

Circuit breakers. If a downstream system returns errors consistently, stop sending requests after N failures. Resume with a health check. This prevents cascading failures and buffer exhaustion.

Checkpointing. After processing each batch or partition, record what was completed. On failure, resume from the last checkpoint. This is the difference between a 5-minute recovery and a 5-hour reprocessing job.

What to Do Next

Map your current pipelines against the four architecture layers. Identify which layers are missing or mixed together. The most common gap: ingestion and transformation are in the same script, making it impossible to replay raw data or isolate failures. Separate them, and reliability follows.

Try Dremio Cloud free for 30 days

Conceptual, Logical, and Physical Data Models Explained

Alex Merced — Wed, 18 Feb 2026 10:00:00 GMT

Most data teams jump straight from a stakeholder request to creating database tables. They skip the planning steps that prevent misalignment, redundancy, and rework. The result: tables that make sense to the engineer who built them but confuse everyone else.

Data modeling addresses this by working at three levels of abstraction. Each level answers a different question, for a different audience, at a different stage of the design process.

Why Three Levels Exist

A single data model can't serve every purpose. Business stakeholders need to see what data the system captures and how concepts relate. Data architects need to define precise structures, data types, and rules. Database engineers need to optimize storage and performance for a specific platform.

Trying to capture all of this in one diagram creates a document that's too abstract for engineers and too technical for the business. Three levels solve this by separating concerns.

The Conceptual Data Model

A conceptual data model defines the big picture. It identifies the major entities your system needs to track and the relationships between them.

For an e-commerce platform, a conceptual model might look like this:

Customer places Order
Order contains Line Item
Line Item references Product
Product belongs to Category

There are no column names, no data types, no keys. The conceptual model exists to answer one question: "Do we agree on what data the system needs?"

This model is created collaboratively with business stakeholders. Its value is alignment. When the finance team says "customer" and the marketing team says "customer," the conceptual model ensures they mean the same thing.

Skip this level, and you build a database that captures the wrong entities or misses key relationships. Fixing structural errors after the database is in production costs 10x more than catching them at conception.

The Logical Data Model

The logical model adds precision to the conceptual model. It defines:

Attributes for each entity (customer_id, customer_name, email, signup_date)
Data types (INTEGER, VARCHAR(255), DATE)
Primary keys (customer_id uniquely identifies each customer)
Foreign keys (order.customer_id references customer.customer_id)
Normalization rules (eliminate redundancy up to Third Normal Form)

The logical model is intentionally DBMS-independent. It works whether you implement it in PostgreSQL, MySQL, Snowflake, or Apache Iceberg tables. This separation matters because it lets you evaluate the design on its own merits before committing to a specific technology.

Normalization is the primary discipline at this level. The logical model eliminates data redundancy by splitting entities into their most atomic forms. A customer's address doesn't live in the orders table — it lives in its own table, referenced by a foreign key.

The Physical Data Model

The physical model translates the logical model into the exact implementation for a specific database engine. This is where theoretical design meets operational reality.

A physical model specifies:

Table names and column definitions (customers, orders, line_items)
Data types specific to the DBMS (BIGINT vs. INTEGER, TIMESTAMP_TZ vs. TIMESTAMP)
Indexes for query performance (B-tree on customer_id, hash on email)
Partitioning strategies (partition orders by order_date using monthly ranges)
Compression and file format choices (Parquet with Snappy compression for Iceberg)

The physical model is where performance tuning happens. You might denormalize at this level — joining the customer name into the orders table to avoid an expensive join at query time — even though the logical model keeps them separate.

In a lakehouse architecture, the physical model also includes Iceberg table properties: partition specs (time-based or value-based), sort orders for query optimization, and file format settings.

How the Three Levels Connect

Each level feeds the next:

Aspect	Conceptual	Logical	Physical
Abstraction	High	Medium	Low
Audience	Business stakeholders	Data architects	Database engineers
Entities	Named	Defined with attributes	Tables with typed columns
Relationships	Named	With cardinality and keys	Foreign key constraints
Data types	None	Generic (INTEGER, VARCHAR)	DBMS-specific (BIGINT, TEXT)
Normalization	Not applicable	Applied (3NF)	May denormalize for performance
Performance	Not considered	Not considered	Indexes, partitions, caching

In platforms like Dremio, you can implement all three levels using virtual datasets organized in a Medallion Architecture. Bronze views represent the physical layer (raw data mapped to typed columns). Silver views represent the logical layer (joins, business keys, normalized relationships). Gold views represent the conceptual layer (business entities ready for consumption, documented with Wikis and tagged with Labels).

Common Mistakes

Skipping the conceptual model. Engineers jump to table creation and miss requirement gaps that surface months later when a stakeholder asks "Why don't we track X?"

Building logical models tied to a DBMS. If your logical model includes PostgreSQL-specific syntax, it's a physical model disguised as a logical one. This makes migration and evaluation harder.

Over-normalizing for analytics. Third Normal Form is correct for transactional systems. But analytics workloads benefit from wider, flatter tables that reduce join counts. Know when to denormalize.

Under-documenting all levels. A model without documentation is a puzzle. Column names like c_id, dt, and amt save keystrokes and cost hours of confusion.

What to Do Next

Audit your current data platform against all three levels. Can you show a business stakeholder what entities your system tracks (conceptual)? Can you show an architect the precise attributes and relationships (logical)? Can you explain why the tables are partitioned and indexed the way they are (physical)?

If any of those questions draws a blank, you have a gap worth filling.

Try Dremio Cloud free for 30 days

How to Build a Semantic Layer: A Step-by-Step Guide

Alex Merced — Wed, 18 Feb 2026 10:00:00 GMT

Most teams start building a semantic layer the wrong way: they open their BI tool, create a few calculated fields, and call it done. Six months later, three dashboards define "churn" differently, nobody trusts the numbers, and the data team is debugging metric discrepancies instead of building new features.

A well-built semantic layer prevents all of that. Here's how to do it right.

Start With Metrics, Not Data Models

Before writing a single line of SQL, sit down with stakeholders from Sales, Finance, Marketing, and Product. Agree on the top 5-10 business metrics your organization uses to make decisions.

For each metric, document:

The calculation: Revenue = SUM(order_total) WHERE status = 'completed' AND refunded = FALSE
The owner: Who is accountable for this definition?
The grain: Daily? Monthly? Per customer?
The refresh cadence: Real-time? Daily batch? Weekly?

This exercise is harder than it sounds. You will discover that "Monthly Active Users" has three competing definitions. That's the point. The semantic layer can't resolve disagreements that haven't been surfaced yet.

Output: A metric glossary. This becomes the source document for everything you build next.

Map Your Data Sources

Inventory every system that feeds into your analytics:

Source Type	Examples	Access Pattern
Transactional databases	PostgreSQL, MySQL, SQL Server	Federated query (read-only)
Cloud data lakes	S3 (Parquet/Iceberg), Azure Data Lake	Direct scan or catalog
SaaS platforms	Salesforce, HubSpot, Stripe	API extraction or replication
Spreadsheets	Google Sheets, Excel	One-time import or scheduled sync

Not all sources need to be replicated into a central store. Federation lets you query data where it lives without the cost and complexity of ETL pipelines. Platforms like Dremio connect to dozens of sources and present them in a single namespace, so your semantic layer can span everything without data movement.

Design the Three-Layer View Structure

The most effective semantic layer architecture uses three layers of SQL views, commonly called the Medallion Architecture.

Bronze Layer (Preparation)

Create one view per raw source table. Apply no business logic. Just make the data human-readable:

Rename cryptic columns: col_7 → OrderDate, cust_id → CustomerID
Cast types to standard formats: strings to dates, integers to decimals
Normalize timestamps to UTC
Avoid using SQL reserved words as column names (Timestamp, Date, Role will force double-quoting in every downstream query. Use EventTimestamp, TransactionDate, UserRole instead.)

Bronze views should be boring. Their only job is to make raw data safe to work with.

Silver Layer (Business Logic)

This is where your metric glossary becomes code. Silver views join Bronze views, deduplicate records, filter invalid data, and apply business rules.

Example:

CREATE VIEW silver.orders_enriched AS
SELECT
    o.OrderID,
    o.OrderDate,
    o.Total AS OrderTotal,
    c.Region,
    c.Segment
FROM bronze.orders_raw o
JOIN bronze.customers_raw c ON o.CustomerID = c.CustomerID
WHERE o.Total > 0 AND o.Status = 'completed';

Each Silver view encodes exactly one business concept. "Revenue" is defined in one place. Every dashboard, notebook, and AI agent that needs revenue queries this view. No exceptions.

Gold Layer (Application)

Gold views are pre-aggregated for specific consumers. A BI dashboard gets monthly_revenue_by_region. An AI agent gets customer_360_summary. A finance report gets quarterly_financial_summary.

Gold views don't add new business logic. They aggregate and reshape Silver views for performance and usability.

Document Everything — or Let AI Help

An undocumented semantic layer is a semantic layer nobody uses. Every table and every column should have a description that explains:

What the data represents
Where it comes from
Any known limitations or caveats

This is tedious work. Modern platforms accelerate it with AI. Dremio's generative AI, for example, can auto-generate Wiki descriptions by sampling table data, and suggest Labels (tags like "PII," "Finance," "Certified") for governance and discoverability. The AI provides a 70% first draft. Your data team fills in the domain-specific context.

This documentation serves two audiences: human analysts browsing the catalog, and AI agents that need context to generate accurate SQL. Both benefit from rich, accurate descriptions.

Enforce Access Policies at the Layer

Security should be embedded in the semantic layer, not applied after the fact in each tool. Two patterns:

Row-Level Security: Filter what data a user can see based on their role. A regional manager sees only their region's data. The SQL view applies the filter automatically.

Column Masking: Mask sensitive columns (SSN, email, salary) for roles that don't need them. Analysts see ****@email.com. Data engineers see the full value.

The advantage of enforcing policies at the semantic layer: every downstream query inherits the rules, whether the query comes from a dashboard, a Python notebook, or an AI agent. No gaps.

Start Small, Then Expand

Don't try to model your entire data landscape on day one. Start with:

3-5 core metrics from your glossary
The 2-3 source systems those metrics depend on
One Bronze → Silver → Gold pipeline per metric

Validate by running the same question across two different tools (a BI dashboard and a SQL notebook, for example). If both return the same number, the semantic layer is working. If they don't, fix the Silver view definition before adding more.

Once the first metrics are stable, expand incrementally. Add new sources, new Silver views, new Gold views. Each addition is low-risk because the layered structure isolates changes.

What to Do Next

Pick the metric your organization argues about the most. Define it explicitly in a Silver view. Test it against the current dashboards. If the numbers match, you've validated the approach. If they don't, you've just found the inconsistency that's been silently costing your organization trust.

Try Dremio Cloud free for 30 days

How to Think Like a Data Engineer

Alex Merced — Wed, 18 Feb 2026 09:00:00 GMT

The median lifespan of a popular data tool is about three years. The tool you master today may be deprecated or replaced by the time your next project ships. What doesn't change are the principles underneath: how data flows, how systems fail, how contracts between producers and consumers work, and how to decompose messy requirements into clean, maintainable pipelines.

Thinking like a data engineer means solving problems at the systems level, not the tool level. It means asking "what could go wrong?" before asking "what framework should I use?"

Tools Change — Principles Don't

Every year brings a new orchestrator, a new streaming framework, a new columnar format. Teams that build their expertise around a specific tool struggle when the landscape shifts. Teams that build expertise around principles — idempotency, schema contracts, data quality at the source, composable stages — adopt new tools without starting over.

The question is never "How do I do this in Tool X?" The question is "What problem am I solving, and what properties does the solution need to have?" Once you answer that, the tool choice becomes a constraint-matching exercise.

The Five Questions Framework

Before designing any pipeline, answer five questions:

1. What data exists? Identify every source: databases, APIs, event streams, files. Note the format (JSON, CSV, Parquet, Avro), volume (rows per day), freshness (real-time, hourly, daily), and reliability (does this source go down?).

2. Where does it need to go? Identify every consumer: dashboards, ML models, downstream systems, analysts. Note what format they need, how fresh the data must be, and what SLAs they expect.

3. What transformations are needed? Map the gap between source shape and consumer shape. This includes cleaning (nulls, duplicates, encoding), enriching (joining lookup data), and aggregating (daily summaries, running totals).

4. What can go wrong? List failure modes: late data, schema changes in the source, duplicate events, null values in required fields, API rate limits, network partitions, out-of-order events. For each failure mode, define the expected behavior — skip, retry, alert, or quarantine.

5. How will you know if it's working? Define observability: row counts in vs. row counts out, freshness checks, schema validation, anomaly detection. If you can't answer this question before building the pipeline, you'll be debugging in production.

Think in Systems, Not Scripts

A script processes data from A to B. A system handles what happens when A is late, B is down, the data shape changes, the volume doubles, and the on-call engineer needs to understand what happened at 3 AM.

Thinking in systems means:

Composability. Break pipelines into discrete stages that can be developed, tested, and monitored independently. An ingestion stage should not also handle transformation and loading. When a stage fails, you restart that stage, not the entire pipeline.

Contracts. Define what each stage produces: column names, data types, value ranges, freshness guarantees. When a producer changes its output, the contract violation is caught immediately — not three stages downstream when a dashboard shows wrong numbers.

State management. Track what has been processed. Know where to resume after a failure. Avoid reprocessing data unnecessarily by maintaining checkpoints, watermarks, or change data capture (CDC) positions.

Isolation. One failing pipeline should not take down others. Shared resources (connection pools, compute clusters, storage) need limits per-pipeline to prevent noisy-neighbor problems.

Design for Failure First

The default assumption should be: every component will fail. Networks drop. APIs return errors. Source schemas change without warning. Storage fills up. The pipeline that handles none of these cases works in development and breaks in production.

Practical failure-first design:

Retry with backoff. Transient errors (network timeouts, API rate limits) often resolve themselves. Retry with exponential backoff before alerting.
Dead-letter queues. Records that can't be processed (malformed, unexpected schema) go to a separate queue for inspection — not dropped silently.
Idempotent writes. Running a pipeline job twice should produce the same end-state. Use upserts, deduplication, or transaction-based writes instead of blind appends.
Circuit breakers. If a downstream system is unresponsive, stop sending data after N failures instead of filling up buffers and crashing.

Anti-Patterns That Signal Inexperience

Choosing the tool before understanding the problem. "We should use Kafka" is not a good starting point. "We need sub-second event delivery with at-least-once guarantees" is. The tool choice follows from the requirements, not the other way around.

Monolithic pipelines. One script that reads from a database, cleans data, joins three tables, aggregates, and writes to a warehouse. When any step fails, the entire pipeline fails. When any step needs a change, the entire pipeline needs retesting.

No error handling. try: process() except: pass is not error handling. Every expected failure mode should have an explicit response: retry, skip and log, alert, or halt.

No monitoring. If the only way you learn about a pipeline failure is when an analyst asks "why is the dashboard empty?", your observability is broken.

What to Do Next

Pick your most critical pipeline. Walk through the Five Questions Framework. Can you answer all five clearly and completely? If not, the gaps are your immediate priorities. Write down the answers, share them with your team, and use them as the specification for your next refactor.

Try Dremio Cloud free for 30 days

What Is Data Modeling? A Complete Guide

Alex Merced — Wed, 18 Feb 2026 09:00:00 GMT

Every database, data warehouse, and data lakehouse starts with the same question: how should this data be organized? Data modeling answers that question by creating a structured blueprint of your data — what it contains, how it relates, and what it means.

A data model is not a diagram you draw once and forget. It's a living definition of your business logic, encoded in the structure of your tables, columns, and relationships. Get it right, and every downstream consumer — dashboards, reports, AI agents, applications — works from the same shared understanding. Get it wrong, and you spend months untangling conflicting definitions of "customer," "revenue," and "active user."

What Data Modeling Actually Means

Data modeling is the process of defining entities, attributes, and relationships for a dataset. Entities represent real-world objects or concepts (Customers, Orders, Products). Attributes describe those entities (customer name, order date, product price). Relationships define how entities connect (a customer places an order, an order contains products).

The goal is to create a representation precise enough that a database can store the data reliably, and clear enough that a human — or an AI agent — can understand what the data means.

Think of it as an architectural blueprint. You wouldn't build a house without one, and you shouldn't build a data platform without a data model.

The Three Levels of Data Modeling

Data models operate at three levels of abstraction, each serving a different audience:

Level	Audience	Purpose	Contains
Conceptual	Business stakeholders	Define what data is needed	Entities, relationships, business rules
Logical	Data architects	Define how data is structured	Attributes, data types, normalization rules, keys
Physical	Database engineers	Define where and how data is stored	Tables, columns, indexes, partitions, constraints

Conceptual models capture business requirements without technical details. A conceptual model might say "Customers place Orders, and Orders contain Products." It doesn't specify column types or index strategies. Its job is to align business stakeholders and technical teams on what data the system needs.

Logical models add precision. They define attributes (customer_id, customer_name, email), assign data types (INTEGER, VARCHAR, TIMESTAMP), and specify normalization rules. A logical model is independent of any specific database engine — it works whether you implement it in PostgreSQL, Snowflake, or Apache Iceberg.

Physical models are implementation-specific. They define table names, column types for a specific DBMS, primary and foreign keys, indexes for query performance, and partitioning strategies. This is where theoretical design meets operational reality — storage formats, compression codecs, and file organization all matter here.

Common Data Modeling Techniques

Several techniques exist for organizing data. Each fits different use cases:

Entity-Relationship (ER) Modeling is the most widely used technique for transactional systems. It maps entities, attributes, and their relationships using formal diagrams. Most OLTP databases — the systems that power applications — start with an ER model.

Dimensional Modeling organizes data into facts (measurable events like sales transactions) and dimensions (context like date, product, and customer). Star schemas and snowflake schemas are the two primary patterns. This technique dominates data warehousing and analytics.

Data Vault Modeling separates data into Hubs (business keys), Links (relationships), and Satellites (descriptive attributes with history). It's designed for environments where sources change frequently and full audit history matters.

Graph Modeling represents data as nodes (entities) and edges (relationships). It's useful when the relationships between data points are as important as the data itself — social networks, recommendation engines, fraud detection.

Why Data Modeling Matters More Than Ever

Three trends have made data modeling more critical, not less:

AI needs structure to be accurate. When an AI agent generates SQL, it relies on well-defined tables, clear column names, and documented relationships. A poorly modeled dataset forces the agent to guess which table contains "revenue" and which join path connects "customers" to "orders." Those guesses create hallucinated queries that return wrong numbers.

Self-service analytics depends on understandable data. Business users exploring data in a BI tool can only self-serve if the data model is intuitive. When tables are named stg_src_cust_v2_final with columns like c1, c2, c3, even experienced analysts give up and file a ticket instead.

Compliance requires traceable definitions. Regulations like GDPR and CCPA demand that organizations know what personal data they store, where it flows, and who can access it. A well-documented data model provides that traceability. Without one, compliance audits turn into archaeology projects.

Platforms like Dremio address this by letting you implement data models as virtual datasets (SQL views) organized in a Medallion Architecture — Bronze for raw data preparation, Silver for business logic and joins, Gold for application-specific outputs. The model exists as a logical layer without requiring physical data copies, and Wikis, Labels, and Fine-Grained Access Control add documentation and governance directly to the model.

What to Do Next

Pick your five most-queried tables. For each one, answer three questions: What does each column mean? How does this table connect to other tables? Who is allowed to see which rows? If you can't answer all three confidently, your data model has gaps.

Filling those gaps means defining clear entities, documenting attributes, and specifying relationships — the core of data modeling.

Try Dremio Cloud free for 30 days

What Is a Semantic Layer? A Complete Guide

Alex Merced — Wed, 18 Feb 2026 09:00:00 GMT

Ask three teams in your company how they calculate "revenue" and you'll get three answers. Sales counts bookings. Finance counts recognized revenue. Marketing counts pipeline value. All three call it "revenue." All three get different numbers. Nobody knows which one is right.

This is the problem a semantic layer solves.

What a Semantic Layer Actually Is

A semantic layer is a logical abstraction between your raw data and the people (or AI agents) querying it. It maps technical database objects — tables, columns, join paths — to business-friendly terms like "Revenue," "Active Customer," or "Churn Rate."

It's not a database. It doesn't store data. It's a layer of definitions, calculations, and context that ensures every query against your data produces consistent results, regardless of which tool or person runs it.

The concept isn't new. Business Objects introduced "universes" in the 1990s — metadata models that let users drag and drop business concepts instead of writing SQL. What's changed is scope. Modern semantic layers are universal (not tied to one BI tool), AI-aware (they provide context to language models), and governance-integrated (they enforce access policies alongside definitions).

What a Semantic Layer Contains

A complete semantic layer includes six components:

Component	What It Does
Virtual datasets (Views)	SQL-defined business logic applied once and reused everywhere
Metric definitions	Canonical calculations for KPIs (e.g., MRR = SUM of active subscription revenue)
Documentation	Human- and machine-readable descriptions of tables, columns, and relationships
Labels and tags	Categorization for governance (PII, Finance) and discovery
Join relationships	Pre-defined join paths so users don't need to know foreign keys
Access policies	Row-level security and column masking enforced at the layer

The key insight: these components serve both human analysts and AI agents. When an AI generates SQL from a natural language question, it consults this same layer to understand what "revenue" means, which tables to join, and which columns to filter.

How It Works in Practice

Here's what happens when someone queries data through a semantic layer:

A user (or AI agent) asks: "What was revenue by region last quarter?"
The semantic layer translates:
- "Revenue" → SUM(orders.total) WHERE orders.status = 'completed'
- "Region" → customers.region
- "Last quarter" → WHERE order_date BETWEEN '2025-10-01' AND '2025-12-31'
The query engine generates optimized SQL against the underlying data sources
Results are returned using business terms, not raw column names

The user never writes SQL. The AI never guesses at column names. The metric definition is applied identically whether the query runs in a dashboard, a Python notebook, or a chat interface.

Why It Matters Now More Than Ever

Three trends are making semantic layers essential, not optional.

AI agents need business context. Large language models generating SQL will hallucinate column names, use incorrect aggregation logic, and join tables wrong unless they have explicit definitions to work from. A semantic layer provides that grounding. This is why platforms like Dremio embed a semantic layer directly into the query engine — it's the context that makes the AI accurate instead of confidently wrong.

Self-service analytics demands accessibility. Business users want to query data without filing a ticket. Exposing raw database schemas to non-technical users creates more problems than it solves. A semantic layer presents data in terms people already understand.

Governance requires centralized definitions. GDPR, CCPA, and industry regulations require organizations to know what data they have, who can access it, and how it's used. A semantic layer centralizes these definitions and enforces access policies in one place instead of across dozens of tools.

Common Misconceptions

"It's just a data catalog." A data catalog is an inventory — it tells you what data exists. A semantic layer defines what data means and how to calculate it. You need both. They're complementary, not interchangeable. (See: Semantic Layer vs. Data Catalog)

"It's just a BI tool feature." Some BI tools include semantic models (Looker's LookML, Power BI's datasets). But these are tool-specific. If your organization uses three BI tools, you maintain three separate semantic models. A universal semantic layer defines metrics once and serves them to every tool.

"It adds a performance penalty." Modern semantic layers don't just translate queries — they optimize them. Dremio, for example, uses Reflections (pre-computed, physically optimized data copies) to accelerate queries that pass through its semantic layer. The result is often faster than querying raw tables directly.

What to Do Next

Pick your organization's five most important metrics. Ask two different teams how each one is calculated. If the answers don't match, that's your signal. You don't have a semantic layer problem — you have a trust problem, and a semantic layer is how you fix it.

Try Dremio Cloud free for 30 days

A 2026 Introduction to Apache Iceberg

Alex Merced — Fri, 13 Feb 2026 09:00:00 GMT

Apache Iceberg is an open-source table format for large analytic datasets. It defines how data files stored on object storage (S3, ADLS, GCS) are organized into a logical table with a schema, partition layout, and consistent point-in-time snapshots. If you've heard the term "data lakehouse," Iceberg is the layer that makes it possible by bringing warehouse-grade reliability to data lake storage.

This post covers what Iceberg is, how its metadata works under the hood, what changed across specification versions 1 through 3, what's being proposed for v4, and how to get started using Iceberg tables with Dremio in about ten minutes.

Where Iceberg Came From

Before Iceberg, most data lake tables used the Hive table format. Hive tracks data by directory paths: one directory per partition, with files inside. That works fine for small tables, but it breaks down at scale. Listing files across thousands of partition directories takes minutes. Schema changes require careful coordination. There's no isolation between readers and writers, so concurrent queries can return inconsistent results.

Netflix hit all of these problems in production around 2017. Ryan Blue and Dan Weeks designed Iceberg to solve them by tracking individual files instead of directories, using file-level metadata instead of a central metastore, and requiring atomic commits for every change. Netflix open-sourced the project, and it entered the Apache Incubator in 2018. By May 2020, Iceberg graduated to an Apache Top-Level Project. Today it's the de facto open table format, adopted by AWS, Google, Snowflake, Databricks, Dremio, Cloudera, and dozens of other vendors.

How Iceberg's Metadata Works

Iceberg replaces directory listings with a tree of metadata files. Each layer in the tree stores progressively finer details about the table's contents.

Catalog Pointer: The catalog (Polaris, Glue, Nessie, or any REST catalog implementation) stores a single pointer to the current metadata file. This is the entry point.

Metadata File (JSON): Contains the current schema, partition specs, sort orders, snapshot list, and table properties. Every write creates a new metadata file and atomically swaps the catalog pointer.

Manifest List (Avro): One per snapshot. Lists all manifest files belonging to that snapshot, along with partition-level summary stats. Query engines use these stats to skip entire manifests that can't match a query's filter.

Manifest Files (Avro): Each manifest tracks a set of data files and stores per-file statistics: file path, partition tuple, record count, and column-level min, max, and null counts. These stats enable file-level pruning during scan planning.

Data Files (Parquet/ORC/Avro): The actual rows, stored in columnar format. Iceberg itself is format-agnostic, though Parquet is the most common choice.

This structure means scan planning is O(1) in metadata lookups rather than O(n) in partition directories. That's the core architectural advantage.

Spec Versions: V1 Through V3 (and V4 Proposals)

Version 1: Analytic Tables (2017–2020)

V1 established the fundamentals: immutable data files, snapshot-based tracking, manifest-level file stats, hidden partitioning, and schema evolution via unique column IDs. Operations were limited to appends and full-partition overwrites.

Version 2: Row-Level Deletes (~2022)

V2 added delete files that encode which rows to remove from existing data files. Position delete files list specific (file, row-number) pairs. Equality delete files specify column values that identify deleted rows. This made UPDATE, DELETE, and MERGE possible without rewriting entire data files. V2 also introduced sequence numbers for ordering concurrent writes and resolving commit conflicts through optimistic concurrency.

Version 3: Extended Capabilities (May 2025)

V3 brought several major additions:

Deletion Vectors: Binary bitmaps stored in Puffin files that replace position deletes. More compact in storage and faster to apply during reads. At most one deletion vector per data file.
Row Lineage: Per-snapshot tracking of row-level identity (first-row-id, added-rows). This enables efficient change data capture (CDC) pipelines directly on Iceberg tables.
New Data Types: variant for semi-structured data, geometry and geography for geospatial workloads, and nanosecond-precision timestamps (timestamp_ns, timestamptz_ns).
Default Values: Columns can specify write-default and initial-default values, making schema evolution smoother.
Multi-Argument Transforms: Partition and sort transforms can accept multiple input columns.
Table Encryption Keys: Built-in support for encrypting data at rest.

Version 4: Active Proposals (2025–2026)

The community is actively discussing several changes for a future v4 spec:

Single-file commits would consolidate all metadata changes into one file per commit, reducing I/O overhead for high-write workloads.
Parquet for metadata would replace Avro-encoded metadata files with Parquet, enabling columnar reads of metadata (only load the fields you need).
Relative path support would store file references relative to the table root, simplifying table migration and replication without metadata rewrites.
Improved column statistics would add more granular stats for better query planning and change detection.

Key Features Worth Knowing

ACID Transactions: Every commit is atomic with serializable isolation. Readers never see partial writes.
Schema Evolution: Add, drop, rename, or reorder columns safely. Iceberg uses unique field IDs, so renaming a column doesn't break older data files.
Partition Evolution: Change your partitioning strategy without rewriting existing data. Old and new partition layouts coexist. Queries filter on data values, not partition columns.
Hidden Partitioning: Users query raw values (WHERE order_date = '2025-06-15'). Iceberg applies transforms (month, day, bucket, truncate) automatically. No synthetic partition columns in the schema.
Time Travel: Query any previous snapshot by ID or timestamp. Roll back to a known-good state in one command.
Branching and Tagging: Named references to specific snapshots, useful for write-audit-publish workflows and staging environments.
Multi-Engine Access: The same Iceberg table is readable and writable from Spark, Flink, Trino, Dremio, DuckDB, Snowflake, BigQuery, Presto, and others.

The Value of the REST Catalog Spec

Iceberg's REST Catalog Specification defines an HTTP API for table management. Any engine that speaks HTTP can create, list, read, and commit to Iceberg tables without importing a Java SDK. That's significant because it makes catalog access language-agnostic (Python, Rust, Go, JavaScript) and cloud-agnostic (AWS, GCP, Azure). It also enables server-side features like credential vending (short-lived storage tokens per request), commit deconfliction, and multi-table transactions.

Several projects implement the REST Catalog spec: Apache Polaris, Project Nessie, Unity Catalog, AWS Glue (via adapter), and Snowflake Open Catalog. This means you can pick a catalog implementation without locking in your query engines. Every engine points at the same REST endpoint.

Getting Started: Apache Iceberg on Dremio

You can get hands-on with Iceberg tables right now using Dremio Cloud. Here's the quick path:

1. Sign up at dremio.com/get-started. You'll get a free 30-day trial. Dremio creates a lakehouse project and an Open Catalog (powered by Apache Polaris) automatically at signup.

2. Create an Iceberg table and insert data:

CREATE FOLDER IF NOT EXISTS db;
CREATE FOLDER IF NOT EXISTS db.schema;

CREATE TABLE db.schema.sales (
  order_id INT,
  customer_name VARCHAR,
  product VARCHAR,
  quantity INT,
  order_date DATE,
  total_amount DECIMAL(10,2)
) PARTITION BY (MONTH(order_date));

INSERT INTO db.schema.sales VALUES
  (1, 'Alice Chen', 'Widget A', 10, '2025-01-15', 150.00),
  (2, 'Bob Smith', 'Widget B', 5, '2025-01-20', 75.00),
  (3, 'Carol Davis', 'Widget A', 8, '2025-02-10', 120.00);

Notice the PARTITION BY (MONTH(order_date)). That's hidden partitioning in action. You query order_date directly; Iceberg handles the partitioning.

3. Query the metadata tables. Dremio exposes Iceberg's metadata through TABLE() functions. These let you inspect the internal state of your table without touching the raw metadata files:

-- View all snapshots (who committed what and when)
SELECT * FROM TABLE(table_snapshot('db.schema.sales'));

-- View commit history
SELECT * FROM TABLE(table_history('db.schema.sales'));

-- View manifest file details
SELECT * FROM TABLE(table_manifests('db.schema.sales'));

-- View partition statistics
SELECT * FROM TABLE(table_partitions('db.schema.sales'));

The table_snapshot query shows each snapshot ID, timestamp, and the operation that created it (append, overwrite, delete). The table_manifests query reveals how many data files and delete files exist in each manifest. Run these after each INSERT or DELETE to see how Iceberg tracks changes internally.

Go Deeper

This post covers the essentials, but Iceberg's spec and ecosystem run deep. If you want the full picture, three books cover the subject end to end:

Apache Iceberg: The Definitive Guide (O'Reilly) by Tomer Shiran, Jason Hughes, and Alex Merced. Free download from Dremio.
Apache Polaris: The Definitive Guide (O'Reilly) by Alex Merced, Andrew Madson, and Tomer Shiran. Free download from Dremio.
Architecting an Apache Iceberg Lakehouse (Manning) by Alex Merced. A hands-on guide to designing modular lakehouse architectures with Spark, Flink, Dremio, and Polaris.

Between these three resources and a free Dremio Cloud trial, you'll have everything you need to build on Apache Iceberg in production.

Dremio Developer Community

Join the Dremio Developer Community Slack Community to learn more about Apache Iceberg, Data Lakehouses and Agentic Analytics.

RAG Isn’t a Modeling Problem. It’s a Data Engineering Problem.

Alex Merced — Tue, 20 Jan 2026 09:00:00 GMT

Get Data Lakehouse Books:

Lakehouse Community:

Retrieval-augmented generation looks deceptively simple.
Embed documents.
Store vectors.
Retrieve context.
Ask an LLM to answer questions.

Early demos reinforce this illusion. A small corpus. Clean documents. Few users. Results look impressive. Many teams conclude that success depends on choosing the right model or the best vector database.

That assumption breaks down fast.

Once RAG systems move into real enterprise environments, progress stalls. Accuracy plateaus. Latency spikes. Answers lose trust. Security teams raise alarms. Engineering teams realize the bottleneck is not the model.

It is the data.

Most organizations do not suffer from a lack of embeddings. They suffer from fragmented data, unclear definitions, inconsistent permissions, and legacy systems never designed for AI access. RAG exposes these weaknesses immediately. It does not hide them.

This is why RAG is turning into a data engineering problem first, and a modeling problem second.

Where RAG Systems Actually Break Down

Enterprise data is messy by default. It lives across warehouses, lakes, SaaS tools, document systems, and operational databases. Each source uses different schemas, naming conventions, and access rules. RAG systems must unify all of it before retrieval even begins.

Data quality issues amplify the problem. Duplicate documents inflate embeddings. Stale records surface outdated answers. Inconsistent metadata makes relevance scoring unreliable. The model retrieves content correctly, but the content itself is wrong.

Governance is the most underestimated failure point. Many RAG pipelines ignore permissions or apply them too late. This creates two bad outcomes. Either the system leaks sensitive data, or engineers restrict access so aggressively that answers become incomplete. Both outcomes erode trust.

Semantic ambiguity adds another layer of friction. Business terms rarely mean one thing. “Revenue,” “active customer,” or “churn” vary by team and context. Vector similarity cannot resolve these differences. Without shared definitions, RAG systems retrieve text, not meaning.

These failures have nothing to do with LLM quality. They stem from weak data foundations.

As a result, teams over-engineer retrieval layers while under-investing in context. They tune indexes. Swap vector databases. Adjust chunk sizes. The core issues remain.

RAG systems succeed when they start with governed, well-defined, and accessible data. When they do not, no amount of modeling innovation compensates for the gap.

Are Vector Databases Over-Engineered for Most Teams?

Vector databases became the default RAG component for a simple reason. They solved a real problem early. Fast similarity search over high-dimensional embeddings was hard to do well. Purpose-built systems filled that gap.

The problem is that the industry quickly treated them as mandatory infrastructure.

For many enterprise use cases, that assumption does not hold. Most RAG workloads do not start at billion-scale embeddings. They start with thousands or tens of thousands of documents. At that scale, established systems like Postgres with pgvector or search engines with vector support perform well enough.

These platforms already exist in most organizations. They are governed. They are monitored. They are understood by operations teams. Adding vector search to them is often cheaper and faster than introducing a new system.

Specialized vector databases still have a role. At large scale, with strict latency requirements and high concurrency, optimized ANN indexes and distributed architectures matter. The tipping point is real. It just arrives later than vendors suggest.

The mistake is not using vector databases. The mistake is leading with them.

When teams optimize the vector layer first, they ignore higher-impact problems. Data duplication. Permission enforcement. Metadata consistency. Hybrid retrieval logic. These issues dominate cost and complexity long before vector search performance does.

Hybrid Search Is the Norm, Not the Exception

Vector search alone is rarely sufficient. Keyword search alone is rarely sufficient. Production RAG systems need both.

Keywords provide precision. Vectors provide semantic recall. Together, they outperform either approach in isolation. This pattern shows up consistently across enterprise deployments.

Despite advances in embedding models, keyword search is not becoming obsolete. Embeddings still struggle with exact matches, rare identifiers, and domain-specific language. They also struggle when the query intent is narrow and literal.

As a result, teams maintain two indexes. One lexical. One vector. They fuse results during retrieval or re-ranking. This adds operational cost, but it improves answer quality.

Some hope that better models will eliminate this complexity. That is unlikely in the near term. Language is both semantic and symbolic. Search systems must reflect that reality.

The practical takeaway is simple. Hybrid retrieval should be an assumption, not an optimization. Architectures that treat vector search as a drop-in replacement for text search fail under real workloads.

Latency Changes Every Design Decision

Real-time RAG systems operate under tight latency budgets. Users expect responses in seconds, not tens of seconds. Retrieval time competes directly with model inference time.

To stay within budget, teams make trade-offs. They cache results. Use approximate search. Reduce embedding size. Retrieve fewer documents. Choose smaller or faster models.

Each choice sacrifices something. Recall. Freshness. Completeness.

The best systems compensate by pushing intelligence closer to the data. Precomputed results. Materialized views. Semantic caching. These techniques reduce work at query time and stabilize performance.

Once again, the bottleneck is not the model. It is the architecture around the data.

The Missing Layer: Semantic Context

Most RAG architectures treat embeddings as context. That is a mistake.

Embeddings capture similarity, not meaning. They do not encode business logic, metric definitions, or governance rules. They do not understand which tables represent the same concept, or which fields are authoritative.

This is where many systems quietly fail. AI agents retrieve text fragments without understanding how those fragments relate. Answers may be syntactically correct but semantically wrong.

A semantic layer changes this dynamic. It provides shared definitions, governed access, and a consistent abstraction over raw data. Instead of retrieving arbitrary documents, AI agents retrieve meaningful concepts.

This reduces ambiguity. It improves trust. It lowers the cognitive load on both users and models.

More importantly, it shifts RAG from document search to reasoning over data.

From RAG Pipelines to Agentic Architectures

As systems evolve, retrieval alone is not enough. AI agents need to ask follow-up questions, call tools, execute queries, and reason across steps.

This requires structured access to data, not just text chunks. It also requires standard interfaces so agents can operate across clients and environments.

Open protocols like MCP reflect this shift. They decouple AI agents from specific tools and allow shared context to be reused across applications. This moves RAG closer to a platform capability than a one-off pipeline.

In this world, the value is not in where vectors live. The value is in how context is defined, governed, and exposed.

Conclusion: Stop Optimizing the Wrong Layer

RAG failures rarely come from weak models. They come from weak data foundations.

Enterprises over-invest in vector infrastructure while under-investing in semantics, governance, and architectural coherence. The result is expensive systems that scale poorly and fail to earn trust.

The most resilient approaches treat RAG as a data platform problem. They start with open storage, shared definitions, hybrid retrieval, and performance optimizations that benefit every workload.

This is where lakehouse-native architectures stand out. Platforms like Dremio focus on unifying data access, enforcing semantics, and accelerating queries across sources without duplication. When AI agents are layered on top of that foundation, retrieval becomes simpler, safer, and faster by default.

As models continue to improve, data problems will remain. Teams that solve for context, not just embeddings, will be the ones that scale AI beyond demos and into durable systems.

A Practical Guide to AI-Assisted Coding Tools

Alex Merced — Thu, 15 Jan 2026 09:00:00 GMT

Get Data Lakehouse Books:

Lakehouse Community:

AI-assisted coding is no longer a novelty. It is becoming a core part of how software gets built.

For years, these tools were easy to describe. They were autocomplete engines. They helped you write boilerplate faster and saved a few keystrokes. Useful, but limited.

That changed quickly.

Over the last two years, large language models gained larger context windows, stronger reasoning, and the ability to use tools. At the same time, AI assistants moved closer to the developer workflow. They gained access to repositories, terminals, build systems, tests, and browsers. What emerged was not just better autocomplete, but something closer to a collaborator.

Today, “AI coding tools” covers a wide range of products. Some live in the terminal and act as autonomous agents. Others are AI-native editors built around chat and planning. Many integrate directly into existing IDEs and quietly assist as you type. Each category solves different problems and comes with different tradeoffs.

This creates confusion for developers trying to make sense of the space. Should you use a CLI agent or an IDE plugin? When does an AI-first editor make sense? How much autonomy is helpful before it becomes risky? And how do pricing, privacy, and workflow fit into the decision?

This blog is a practical guide to that landscape. We will categorize the major types of AI-assisted coding tools, compare how they work, and explain when each approach makes sense. The goal is not to crown a single “best” tool, but to give you a clear mental model for choosing the right one for your work.

The Core Taxonomy of AI Coding Tools

Before comparing individual products, it helps to understand how these tools differ at a structural level. Most confusion in this space comes from treating all AI coding tools as the same thing. They are not.

There are three dimensions that matter most: how you interact with the tool, where it runs, and how much autonomy it has.

Interaction Model

Some tools are designed to assist while you type. These focus on inline suggestions and small edits. You stay in control at all times, and the AI reacts to your actions.

Others are chat-driven. You describe what you want in natural language, and the tool responds with explanations, code snippets, or suggested changes. These are useful for learning, debugging, and reasoning about unfamiliar code.

The newest category is agent-based. These tools accept a goal, break it into steps, and execute those steps across files and tools. They plan, act, and revise, often with minimal input once started.

Execution Surface

Where a tool lives shapes how powerful it can be.

Terminal-based tools operate directly on your filesystem and development tools. They can run tests, modify many files, and integrate naturally with scripting and automation workflows.

IDE-native editors are built around AI as a first-class concept. They blend editing, chat, execution, and preview into a single environment designed for iterative work with an assistant.

IDE plugins integrate into existing editors. They trade raw power for familiarity and low friction. You get help without changing how you work.

Browser-based tools prioritize accessibility and collaboration but are usually more constrained in what they can access or modify.

Autonomy Spectrum

Not all AI tools act independently.

Some only suggest. You decide what to accept.

Some perform tasks but wait for confirmation before each step.

Others operate with high autonomy. They plan multi-step changes, run commands, and verify results before handing control back to you.

More autonomy can mean more leverage. It also means more responsibility. Understanding where a tool sits on this spectrum is critical for using it safely and effectively.

With these dimensions in mind, the rest of the landscape becomes much easier to navigate. Each tool is a different point in this design space, optimized for different types of work and different levels of trust.

Terminal-Based AI Coding Agents

Terminal-based AI coding agents are the most powerful and, at times, the most intimidating tools in this space. They live where your code actually runs. That gives them capabilities that IDE plugins cannot match.

Instead of suggesting code, these tools operate directly on your project. They can read files, modify directories, run tests, execute build commands, and interact with version control. In practice, this means they behave less like autocomplete and more like junior engineers following instructions.

Why Terminal Agents Exist

The terminal is already the control plane for software development. It is where builds run, tests fail, migrations execute, and deployments start. By placing AI here, these tools gain first-class access to the real workflow rather than a simulated one.

This makes them well-suited for tasks that span many files or steps. Examples include refactoring large codebases, fixing failing test suites, scaffolding new services, or migrating configurations. These are jobs that are slow and error-prone when done manually.

Representative Tools

Tools in this category include Claude Code, Gemini CLI, OpenCode, and Qodo CLI. While they differ in implementation, they share common traits.

They accept high-level goals instead of line-level instructions. They reason about the repository as a whole. They can chain actions together without repeated prompting. Many of them support approval checkpoints so you can review actions before execution.

Some focus on being general-purpose agents. Others emphasize customization, allowing teams to define their own agents for reviews, testing, or compliance checks.

Strengths and Tradeoffs

The strength of terminal agents is leverage. A single prompt can replace dozens of manual steps. They are especially effective for backend, infrastructure, and data engineering work, where tasks are procedural and tool-driven.

The tradeoff is risk. These tools can change many files quickly. They can run commands that alter state. Used carelessly, they can introduce subtle bugs or destructive changes.

Best practice is to treat terminal agents as powerful automation tools. Keep them scoped. Review diffs. Use version control aggressively. Start with low autonomy and increase it only when trust is earned.

Terminal-based agents are not for every developer or every task. But when used well, they represent one of the biggest productivity jumps in modern software development.

AI-Native IDEs and Editors

AI-native IDEs are built around the assumption that an assistant is always present. Instead of adding AI as a feature, these tools redesign the editor itself to make planning, execution, and iteration flow through the model.

This changes how development feels. You do not switch between typing code and asking for help. The conversation and the code evolve together.

What Makes an IDE AI-Native

In an AI-native IDE, the assistant has persistent awareness of the project. It understands file structure, dependencies, and recent changes without being reminded each time.

These editors usually combine several capabilities in one place. You can ask the assistant to design a feature, generate code across files, run the application, and inspect the results. Some can open a browser, preview a UI, or analyze logs as part of the same workflow.

Another defining trait is planning. The assistant often explains what it is going to do before doing it. This makes complex changes easier to reason about and review.

Representative Tools

Examples in this category include Cursor, Windsurf, Antigravity, and Zed.

Cursor extends the familiar VS Code experience with deep repository understanding and large-scale refactoring capabilities. Windsurf emphasizes agent-driven workflows that keep developers in flow. Antigravity pushes further into full agent autonomy, allowing models to plan, build, and verify changes using integrated tools. Zed focuses on speed, collaboration, and predictive editing, blending performance with AI assistance.

While their design philosophies differ, all of them treat AI as a core part of the editing experience rather than an add-on.

When an AI-Native IDE Makes Sense

These tools shine when you are building features end to end. They work well for rapid prototyping, greenfield projects, and iterative product development.

They are also a good fit for solo developers or small teams, where context switching is expensive and speed matters more than strict process. For some developers, they can replace multiple tools with a single environment.

The downside is commitment. Adopting an AI-native IDE often means changing editors or workflows. For teams with established tooling or strict policies, that may be a barrier.

When the fit is right, though, AI-native IDEs offer a glimpse of what development looks like when the assistant is not a helper, but a constant collaborator.

AI Assistants Embedded in Traditional IDEs

Not every developer wants to change editors or rethink their workflow. For many teams, the most practical entry point into AI-assisted coding is through tools that integrate directly into existing IDEs.

These assistants focus on augmentation rather than replacement. They enhance familiar environments with AI capabilities while preserving established habits, shortcuts, and extensions.

The Copilot Model

This category is defined by inline assistance. The AI observes the code you are writing and offers suggestions in real time. You remain in control, accepting or rejecting changes as you go.

Most tools in this group also include a chat interface. This allows you to ask questions about your code, request explanations, generate tests, or debug errors without leaving the editor. The interaction is conversational, but the execution remains manual.

The emphasis is on incremental gains. These tools aim to make each coding session smoother rather than automate entire tasks.

Representative Tools

GitHub Copilot is the most well-known example. Others include Amazon CodeWhisperer and Amazon Q Developer, JetBrains AI Assistant, Tabnine, and Replit Ghostwriter.

These tools support a wide range of IDEs such as VS Code, JetBrains products, and browser-based environments. They tend to work across many programming languages and frameworks, making them broadly applicable.

Some lean toward individual productivity. Others emphasize enterprise features like policy enforcement, auditability, and security scanning.

Strengths and Limitations

The biggest strength of IDE-embedded assistants is low friction. Developers can adopt them with minimal change and see immediate benefits. They are well suited for day-to-day coding, learning new APIs, and reducing repetitive work.

Their limitation is scope. They usually do not plan or execute multi-step changes on their own. They lack direct access to the terminal and external tools, which limits their autonomy.

For many teams, this is a feature, not a flaw. Embedded assistants provide a safe, predictable way to bring AI into the development process without surrendering control.

They are often the right choice when consistency, governance, and gradual adoption matter more than maximum automation.

Pricing Models and Economic Tradeoffs

AI-assisted coding tools vary widely in their pricing. Understanding these models is important, because cost often scales with autonomy, context size, and usage intensity.

What looks inexpensive at first can become costly at scale. What looks expensive may replace significant engineering time.

Common Pricing Patterns

One common approach is free or freemium access for individuals. These tiers usually offer limited usage, smaller context windows, or restricted agent capabilities. They are designed to encourage experimentation and personal use.

Another model is flat monthly subscriptions per developer. This is common for IDE plugins and AI-native editors. In exchange for a predictable cost, you get higher usage limits, access to stronger models, and better performance.

Agentic tools often introduce credit-based pricing. Each task or action consumes credits based on model usage, context size, and tool execution. This aligns cost with work performed but requires more monitoring.

Enterprise plans layer on governance features. These include audit logs, centralized billing, access controls, and private deployments. Pricing here reflects not just usage, but risk reduction and compliance.

Cost vs Capability Tradeoffs

More powerful tools cost more because they do more. Large context windows, multi-file reasoning, and autonomous execution all increase compute usage.

Autocomplete-focused tools are usually the cheapest. Agent-based systems are the most expensive, especially when used heavily.

Another factor is model flexibility. Tools that allow you to bring your own API keys shift costs directly to the underlying model provider. This can be cheaper or more expensive depending on how you use them.

The right question is not “which tool is cheapest,” but “which tool replaces the most manual effort for my work.”

Individual vs Team Economics

For individuals, free tiers and modest subscriptions often deliver outsized value. Even small time savings justify the cost.

For teams, the equation changes. A tool that saves minutes per developer per day may justify its cost. One that automates entire workflows may justify much more, but only if guardrails are in place.

Understanding pricing early helps avoid mismatches between expectations, usage, and budget. AI tools are productivity multipliers, but only when their costs align with how they are used.

Workflow Patterns Enabled by AI Coding Tools

The real impact of AI-assisted coding is not in individual features, but in how workflows change. Once these tools are part of daily work, the structure of development itself begins to shift.

Instead of writing everything by hand, developers increasingly describe intent, review outcomes, and refine results.

Common AI-Driven Workflows

One of the most common patterns is assisted implementation. Developers sketch function signatures or write descriptive comments, then let the AI fill in the logic. This is especially effective for boilerplate, data transformations, and repetitive patterns.

Debugging is another strong use case. AI tools can explain error messages, trace logic across files, and suggest fixes based on context. This reduces time spent searching documentation or past issues.

Test and documentation generation have also become routine. Many teams now generate unit tests, integration tests, and API docs as part of normal development, not as an afterthought.

Agentic Workflows

Agentic tools enable workflows that were previously impractical.

A single prompt can scaffold a new service, refactor an entire module, or migrate configurations across environments. The agent plans the steps, applies changes, and verifies results before returning control.

These workflows work best when tasks are well-scoped and repeatable. Infrastructure changes, dependency upgrades, and large-scale refactors are strong candidates.

The key is oversight. Developers define the goal and constraints, then review the agent’s output carefully. Agentic workflows reward clarity and discipline.

Shifting the Role of the Developer

As AI takes on more mechanical work, the developer’s role shifts toward design, review, and decision-making.

Time moves away from syntax and toward intent. Understanding systems and tradeoffs becomes more valuable than memorizing APIs.

Teams that adapt their workflows intentionally see the biggest gains. Those that treat AI as a novelty often see uneven results.

AI does not remove the need for good engineering practices. It amplifies them.

Skills Developers Need in the AI Coding Era

AI-assisted coding changes what it means to be effective as a developer. The most valuable skills are shifting away from speed of typing and toward clarity of thinking.

Using these tools well is not about tricks. It is about communication, judgment, and system-level understanding.

Prompting as Specification

Prompting is best understood as writing specifications in natural language.

Clear prompts describe intent, constraints, and context. Vague prompts produce vague results. The best outcomes come from treating the AI like a teammate who needs good requirements.

Effective developers iterate. They refine prompts based on output, correct assumptions, and narrow scope. This feedback loop is fast, but it still requires attention.

Review and Verification

AI-generated code must be reviewed like any other contribution.

Developers need to read diffs carefully, understand the logic, and verify behavior with tests. Blind trust leads to subtle bugs and security issues.

Knowing how to ask the AI to explain its choices is a useful verification technique. If the explanation does not make sense, the code likely does not either.

System Thinking and Constraints

AI tools are strongest when they understand the system they are working in.

Developers who can explain architecture, performance constraints, and operational requirements get better results. This includes knowing what not to automate.

The more autonomy a tool has, the more important boundaries become. Skilled developers define those boundaries clearly.

In the AI coding era, judgment matters more than ever. The tools move fast. It is the developer’s responsibility to steer them well.

Security, Privacy, and Governance Considerations

As AI coding tools gain access to repositories, terminals, and infrastructure, security and governance move from secondary concerns to first-order design questions.

The risks are not hypothetical. These tools can read proprietary code, modify critical systems, and generate output that looks correct but is not.

Code and Data Exposure

Most AI tools rely on remote models. This means code or prompts may leave your local environment.

Developers and teams must understand what data is sent, how long it is retained, and whether it is used for training. Some tools explicitly guarantee no training on customer code. Others allow opt-outs or require enterprise agreements.

For sensitive environments, tools that support local models or on-prem deployment reduce exposure. This often comes at the cost of convenience or model quality.

Autonomy and Guardrails

Agentic tools increase risk by design. They can execute commands, modify configurations, and affect production systems.

Guardrails are essential. These include confirmation prompts, restricted permissions, read-only modes, and sandboxed environments. Version control is a non-negotiable safety net.

The goal is not to eliminate autonomy, but to scope it carefully.

Organizational Governance

For teams, governance features matter as much as raw capability.

Audit logs, access controls, usage monitoring, and policy enforcement help organizations understand how AI tools are being used. They also help prevent accidental misuse.

Clear guidelines reduce risk. Teams should define which tools are allowed, what data they can access, and what level of autonomy is acceptable.

AI-assisted coding can be safe and effective. It requires intentional design, not blind adoption.

How to Choose the Right Tool for You

With so many options, choosing an AI coding tool can feel overwhelming. The key is to match the tool to your role, environment, and tolerance for change.

There is no universal best choice. There is only what fits your work.

By Role

Solo developers often benefit from AI-native IDEs or terminal agents. These tools reduce context switching and accelerate end-to-end work. They are well suited for prototyping, side projects, and greenfield development.

Backend and platform engineers often gain the most from terminal-based agents. These tools align naturally with scripting, automation, and infrastructure tasks.

Frontend and product-focused developers may prefer AI-native editors or IDE plugins that emphasize iteration, previews, and refactoring.

Teams working in large codebases often start with IDE-embedded assistants. These tools improve productivity without disrupting existing processes.

By Environment

Startups and small teams can afford to experiment. Speed and leverage matter more than strict controls, making agentic tools attractive.

Enterprises prioritize predictability and governance. Tools with clear data policies, audit logs, and controlled autonomy are easier to adopt.

Highly regulated environments may require on-prem models or strict data isolation. This narrows the field but reduces risk.

By Autonomy and Trust

If you are new to AI-assisted coding, start with tools that suggest rather than act. Build intuition and confidence before increasing autonomy.

As trust grows, introduce agents for well-scoped tasks. Avoid full autonomy in critical systems until guardrails are proven.

The best choice is one that fits your current needs and can evolve with your workflow. AI tools are not static. Your adoption strategy should not be either.

The Future of AI-Assisted Coding

AI-assisted coding is still early, but the direction is clear. These tools are moving from helpers to participants in the development process.

The distinction between editor, assistant, and agent is already starting to blur.

Convergence of Tools

IDE plugins are gaining agentic capabilities. Terminal agents are adding richer interfaces. AI-native IDEs are absorbing features from both.

Over time, the market will likely converge around flexible systems that can operate at different levels of autonomy depending on context. One tool may act as an autocomplete engine in one moment and an autonomous agent in the next.

Interoperability and Protocols

As AI tools grow more capable, interoperability becomes essential.

Standards for tool access, context sharing, and action execution are emerging. These allow models to interact with editors, terminals, and external systems in consistent ways.

This reduces lock-in and makes it easier to mix tools, models, and workflows.

AI as a First-Class Team Member

The long-term shift is conceptual.

AI tools are evolving from passive assistants into collaborators that can plan work, execute tasks, and verify results. This does not remove the need for human developers. It changes where their effort is spent.

Design, judgment, and accountability remain human responsibilities. Execution increasingly becomes shared.

The future of software development is not fully automated. It is more leveraged, more intentional, and more collaborative.

Sample Prompts to Get Started

One of the hardest parts of using AI coding tools for the first time is knowing what to ask. The prompts below are designed to be simple, low-risk, and useful across most tools, whether you are using a terminal agent, an AI-native IDE, or an IDE plugin.

Each prompt focuses on building or modifying something small while helping you learn how the tool behaves.

Prompt 1: Create a Simple Project Skeleton

Use this to test repo awareness and file creation.

Create a simple Python project for a command-line tool.

It should include:

A README

A main entry file

A basic argument parser

Do not add extra features.

This prompt helps you see how the tool structures files and how much initiative it takes.

Prompt 2: Implement a Small Feature From a Description

Use this to test code generation quality.

Add a function that reads a CSV file and prints the top 5 rows.

Assume the file path is passed as a command-line argument.

This works well in IDE plugins and editors. Review the code carefully and run it.

Prompt 3: Explain Existing Code

Use this to test understanding and explanation.

Explain what this function does and identify any edge cases.

Keep the explanation concise.

This is useful for learning unfamiliar code and validating AI understanding.

Prompt 4: Generate Tests

Use this to test correctness and coverage.

Write unit tests for this function.

Use the existing testing framework.

Cover normal cases and one edge case.

This helps establish a review habit and reinforces test-driven thinking.

Prompt 5: Refactor for Clarity

Use this to test refactoring behavior.

Refactor this code to improve readability.

Do not change behavior.

Keep the logic explicit.

Compare the diff to ensure intent is preserved.

Prompt 6: Simple Agentic Task (Terminal or AI-Native IDE)

Use this to test safe autonomy.

Add basic logging to this application.

Use the existing logging library.

Show me the changes before committing.

This prompt checks whether the agent plans steps and respects boundaries.

Prompt 7: Debug a Failure

Use this to test reasoning.

This test is failing.

Explain why, then propose a fix.

Do not apply the fix yet.

Only apply changes after reviewing the explanation.

How to Use These Prompts Safely

Start small. Run tools in a clean project or branch. Review every change.

Pay attention to how the tool interprets ambiguity. If results are surprising, refine the prompt rather than forcing acceptance.

Good prompts are clear, scoped, and explicit about constraints. Treat them like lightweight specifications.

These examples are not about speed. They are about learning how the tool thinks before trusting it with more responsibility.

Conclusion

AI-assisted coding is no longer a single category of tools. It is an ecosystem with distinct approaches, tradeoffs, and philosophies.

Terminal agents offer raw power and automation. AI-native IDEs rethink how development flows. IDE-embedded assistants provide steady gains with minimal disruption. Each has a place, and each serves different kinds of work.

The most important takeaway is intentionality. The value of these tools depends less on which one you choose and more on how you use it. Clear goals, strong review practices, and appropriate guardrails matter more than novelty.

AI does not replace good engineering. It rewards it.

Developers who understand their systems, communicate intent clearly, and exercise judgment will see the greatest benefit. Those who treat AI as a shortcut risk confusion and fragility.

The opportunity is significant. Used well, AI-assisted coding can reduce toil, accelerate learning, and free time for higher-level thinking. The tools are ready. The challenge now is

Building Pangolin - My Holiday Break, an AI IDE, and a Lakehouse Catalog for the Curious

Alex Merced — Thu, 15 Jan 2026 09:00:00 GMT

Get Data Lakehouse Books:

Lakehouse Community:

1. Introduction: A Holiday, an Agent, and an Idea

In December 2025, Google released something that changed how I code—Antigravity IDE. It wasn’t just another brilliant editor. It came packed with AI agents that could write code, test it, refactor it, and even debug alongside you. Naturally, I had to try it out.

I didn’t jump right into building a big project. Instead, I used it to make some tooling for Dremio and Apache Iceberg, both technologies I work with frequently. That experience set the foundation for something bigger: Pangolin, an open-source, feature-rich lakehouse catalog.

This blog tells the story of how Pangolin came to be. It’s not a pitch for production use. It’s a working concept, a glimpse into what’s possible.

2. First Steps: Learning to Trust the Agent

Before Pangolin, I started small. I needed to understand how to work with the Antigravity coding agent in a way that felt predictable and collaborative. So I created four tools:

dremioframe: A DataFrame-style API for building Dremio SQL queries in Python.
iceframe: A similar API, but for building Iceberg-compatible queries using local compute.
dremio-cli: A command-line tool for interacting with Dremio.
iceberg-cli: A CLI that filled in the gaps left by pyiceberg.

These tools weren’t just functional; they became learning tools. I practiced writing clear prompts, specifying inputs and outputs, and most importantly, asking the agent to generate and refine unit, live, and regression tests. I also got better at pushing back when something didn’t work.

Once I felt confident in that workflow, writing specs, prompting the agent, challenging assumptions, and getting results, I was ready to build something bigger.

3. Rethinking the Lakehouse Catalog

Catalogs are central to the Iceberg ecosystem. They’re how engines discover, manage, and track tables. But most catalogs out there either focus on infrastructure or metadata—not both.

Some great projects inspired Pangolin:

Project Nessie: Created at Dremio, Nessie brought Git-like versioning to data catalogs. It’s a brilliant idea that still powers tools like Bauplan. But Nessie doesn’t support features like multi-tenancy or catalog federation.
Apache Polaris: Polaris, co-created by Dremio and Snowflake and now an Apache Incubator project, is well on its way to becoming the open standard. It supports RBAC, catalog federation, generic assets, and upcoming table services that proxy metadata processing.
Business metadata platforms (DataHub, Atlan, Collibra, etc.): These tools focus on discovery and access workflows, and some now support Iceberg. But they bolt onto a catalog—they don’t start as one.

That got me thinking: What if a single open source catalog could do it all?

Pangolin became my experiment to find out.

4. Feature List: The Dream Catalog

Before writing a single line of code, I wrote down everything I wanted this catalog to do, the features I admired in other tools, the gaps I noticed, and a few experiments I just wanted to try.

Here’s what ended up on the list:

Catalog versioning, with support for branching and merging, but scoped-branches don't have to affect all tables.
Catalog federation, so one catalog can reference others.
Generic asset support, to register Delta tables, CSV datasets, or even external databases alongside Iceberg tables.
Business metadata, including access requests and grant workflows.
Multi-tenancy, so each team can work in its own isolated space.
RBAC and TBAC (tag-based access control), to control access based on roles and tags.
No-auth mode, to make it easy to spin up and test locally.
Credential vending, with built-in support for AWS, Azure, GCP, and S3-compatible systems.
Pluggable backends, starting with PostgreSQL and MongoDB for metadata persistence.

That’s a lot. But I didn’t set out to build a polished product—I just wanted to see if it was possible.

5. Choosing the Stack: Why Rust, Python, and Svelte

With the feature list in hand, the next decision was the tech stack. I know Python and JavaScript like the back of my hand, which would’ve made it easy to move fast. But I wanted something that would scale better—and maybe be a little less error-prone.

I considered three languages for the backend: Java, Go, and Rust.

Java is the standard in the data world. But writing clean, scalable Java means understanding the JVM inside and out. I know it—but not enough to move quickly.
Go is simple and efficient. Rust is strict and safe. Between the two, I picked Rust.

Rust’s compiler errors are frustrating at first but turn into a superpower. The strong typing and detailed feedback also pair well with AI agents; errors are easier to reason about and fix through prompting.

For the rest of the stack:

Rust powers the backend and CLI.
Python powers the SDK and scripting layer.
Svelte powers the UI—lightweight and reactive, but more complex than I expected once the feature count grew.

All in, I ended up with a full stack that balanced experimentation and real-world usability. The only problem was... building it all over a holiday break.

6. Building It: 100 Hours, Three Interfaces, and a Lot of Feedback Loops

Once I committed to the stack, the pace picked up fast. I spent roughly 100 hours on Pangolin, which ended up taking most of my holiday break. The backend came together first, followed by the Rust-based CLI and then the Python SDK.

The backend covered all the core ideas: catalogs, tenants, assets, access rules, and credential vending. Rust helped here. The compiler forced clarity. Each time something felt vague, the type system pushed back until the design made sense.

The Python SDK turned out better than I expected. It didn’t just wrap the API. It made some features practical. Generic assets are a good example. Through the SDK, those assets became usable for sharing database connections, Delta tables, CSV datasets, and other non-Iceberg data without much friction.

The hardest part was the UI.

With so many features, state management became tricky fast. I used Antigravity’s browser agent early on, and it helped catch basic issues. Once the UI grew more complex, manual testing worked better. I spent a lot of time clicking through edge cases, capturing network requests, reading console errors, and feeding that context back to the agent. It was slower, but it worked.

By the end, Pangolin had three real interfaces: a Rust CLI, a Python SDK, and a Svelte UI. All of them worked against the same API and feature set.

7. What Pangolin Is—and What It Isn’t

Pangolin exists. You can run it. You can click around, create catalogs, register assets, request access, and vend credentials across clouds.

That said, I don’t see Pangolin as a production catalog. I don’t plan to invest heavily beyond bug fixes and minor improvements. For a truly open, production-ready lakehouse catalog, Apache Polaris is still the best option today. If you want a managed path, platforms like Dremio Catalog, which build on Polaris, handle the complex parts for you.

Pangolin serves a different purpose. It’s a proof of concept. It shows what can happen when a community-oriented project tries to bring versioning, federation, governance, business metadata, and access workflows together in one place.

If you’re a lakehouse nerd like me, Pangolin might be fun to explore. If it sparks ideas or nudges other projects to co-locate these features sooner, then it did its job.

Make sure to follow me on linkedin and substack

What Are Recursive Language Models?

Alex Merced — Sat, 10 Jan 2026 09:00:00 GMT

Get Data Lakehouse Books:

Lakehouse Community:

Recursive Language Models (RLMs) are language models that call themselves.

That sounds strange at first—but the idea is simple. Instead of answering a question in one go, an RLM breaks the task into smaller parts, then asks itself those sub-questions. It builds the answer step by step, using structured function calls along the way.

This is different from how standard LLMs work. A typical model tries to predict the full response directly from a prompt. If the task has multiple steps, it has to manage them all in a single stream of text. That can work for short tasks, but it often falls apart when the model needs to remember intermediate results or reuse the same logic multiple times.

RLMs don’t try to do everything at once. They write and execute structured calls—like CALL("question", args)—inside their own output. The system sees this call, pauses the main response, evaluates the subtask, then inserts the result and continues. It’s a recursive loop: the model is both the planner and the executor.

This gives RLMs a kind of dynamic memory and control flow. They can stop, plan, re-enter themselves with new input, and combine results. That’s what makes them powerful—and fundamentally different from the static prompting methods most models use today.

What Problem Do RLMs Solve?

Language models are good at sounding smart. But when the task involves multiple steps, especially ones that depend on each other, standard models often fail.

Why? Because they generate everything in a straight line.

If you ask a regular LLM to solve a logic puzzle, it has to juggle the entire solution in one pass. There’s no mechanism to stop, break the task apart, and reuse parts of its own reasoning. It has no structure—just one long stream of text.

Prompt engineering helps, but only up to a point. You can ask the model to “think step by step” or “show your work,” and that can improve results. But these tricks don’t change how the model actually runs. It still generates everything in one session, with no built-in way to modularize or reuse logic.

Recursive Language Models change this. They treat complex tasks as programs. The model doesn’t just answer—it writes code-like calls to itself. Those calls are evaluated in real time, and their results are folded back into the response.

This lets RLMs:

Reuse their own logic.
Focus on one part of the task at a time.
Scale to deeper or more recursive problems.

In other words, RLMs solve the structure problem. They bring composability and control into language generation—two things that most LLMs still lack.

How Do RLMs Actually Work?

At the core of Recursive Language Models is a simple but powerful loop: generate, detect, call, repeat.

Here’s how it plays out:

The model receives a prompt.
It starts generating a response.
When it hits a subtask, it emits a structured function call—something like CALL("Summarize", "text goes here").
The system pauses, evaluates that call by feeding it back into the same model, and gets a result.
The result is inserted, and the original response resumes.

This process can happen once—or dozens of times inside a single response.

Let’s take a concrete example. Suppose you ask an RLM to explain a complicated technical article. Instead of trying to summarize the whole thing at once, the model might first break the article into sections. Then it could issue recursive calls to summarize each section individually. After that, it could combine those pieces into a final answer.

So what’s actually new here?

The model isn’t just generating text. It’s controlling execution.
Each function call is explicit and machine-readable. It’s not hidden in plain text.
The model learns not just what to say, but when to delegate subtasks to itself.

This design introduces modular reasoning. It’s closer to programming than prompting. And it’s what makes RLMs capable of solving longer, deeper, and more compositional tasks than traditional LLMs.

How Are RLMs Different From Reasoning Models?

It’s easy to confuse Recursive Language Models with models designed for reasoning. After all, both aim to solve harder, multi-step problems. But they take very different paths.

Reasoning models try to think better within a fixed response. They rely on prompting tricks (“Let’s think step by step”), fine-tuning, or architectural tweaks to encourage more logical answers. But they still generate their full output in one go. There’s no built-in structure or recursion—just better text generation.

Recursive Language Models go further. They change how language models run, not just how they think.

Here’s the key distinction:

Reasoning models operate in a flat, linear space. They can simulate step-by-step thinking, but they don’t control execution.
RLMs introduce a real control flow. They can pause, emit a sub-call, re-enter themselves, and build results incrementally.

Think of it this way: reasoning models try to write better essays. RLMs write and run programs.

This also makes RLMs easier to inspect and debug. Each recursive call is explicit. You can see the full tree of operations the model performed—what it asked, what it answered, and how it combined the results. That transparency is rare in LLM workflows, and it opens the door to more robust systems.

So while reasoning models stretch the limits of static prompting, RLMs redefine what a model can do at runtime.

Why Recursion Changes What LLMs Can Do

Recursion isn’t just a technical upgrade—it’s a shift in what language models are capable of.

With recursion, models don’t have to guess the whole answer in one pass. They can build it piece by piece, reusing their own capabilities as needed. This unlocks new behaviors that standard models struggle with.

Here’s what that looks like in practice:

Logic puzzles: Instead of brute-forcing a full solution, an RLM can write out each rule, evaluate sub-cases, and combine the results.
Math word problems: The model can break a complex problem into steps, solve each one recursively, and verify intermediate answers.
Code generation: RLMs can draft a function, then call themselves to write test cases, fix bugs, or generate helper functions.
Proof generation: For theorem proving, recursion lets the model build a proof tree, checking smaller lemmas along the way.

In the paper’s experiments, RLMs outperformed non-recursive baselines on multi-step benchmarks. They were also more efficient. Recursive calls reduced total token usage, because the model could reuse logic instead of repeating it .

This is a key point: recursion isn’t just about accuracy. It’s also about efficiency and composability. Instead of scaling linearly with problem size, RLMs can scale logarithmically by solving smaller pieces and reusing solutions.

That makes them a better fit for tasks where reasoning depth grows quickly—exactly the kind of problems LLMs are starting to face in real-world applications.

Why This Matters Now

Language models are everywhere—but most still follow a simple pattern: input goes in, output comes out. That’s fine for quick answers or lightweight tasks. But for anything complex, it’s not enough.

Today, developers are building agents, chains, and tool-using systems on top of LLMs. These wrappers simulate structure, but they’re often fragile. They rely on prompt hacking, regex parsing, and external orchestration to manage what the model can’t do natively.

Recursive Language Models offer a cleaner path. Instead of bolting on structure from the outside, they build it in.

This matters for a few reasons:

Fewer moving parts: RLMs remove the need for external chains or custom routing logic. The model decides when and how to branch.
Greater transparency: Each recursive call is visible and traceable. You can audit what the model did, step by step.
Better generalization: Once trained to use recursion, the model can apply it flexibly across domains—math, code, reasoning, even planning.

And we’re just getting started. RLMs are early, but they hint at a broader shift: treating models not just as generators, but as runtime environments. That opens the door to future systems where models can plan, act, and adapt on their own, with clear structure behind every step.

If the last few years were about making LLMs sound smart, the next few might be about making them think with structure. That’s where recursion fits in.

Conclusion: A New Way to Think with Language Models

Recursive Language Models aren’t just a tweak to existing LLMs. They represent a shift in how models operate.

Instead of treating every task as a one-shot prediction, RLMs break problems into parts, solve them recursively, and combine the results. That gives them something most language models still lack: structure.

This structure matters. It makes models more reliable on complex tasks. It makes their reasoning easier to follow. And it opens the door to new capabilities—like planning, verifying, or adapting—without needing complex external systems.

We’re still early in this space. But the idea is simple and powerful: give models the tools to use themselves. From there, a new class of language systems can emerge—not just fluent, but recursive, modular, and built to handle depth.

RLMs don’t just make better answers. They make better models.

2025 Year in Review Apache Iceberg, Polaris, Parquet, and Arrow

Alex Merced — Mon, 29 Dec 2025 09:00:00 GMT

Get Data Lakehouse Books:

Lakehouse Community:

The open lakehouse is no longer a concept. In 2025, key Apache projects matured, making data warehouse performance on object storage a practical reality. This post walks through the most critical developments in four of those projects: Iceberg, Polaris, Parquet, and Arrow. Each is building a critical layer for an open, engine-agnostic analytics stack.

We start with Iceberg.

Apache Iceberg

Iceberg spent 2025 delivering the core elements of Format Version 3, while setting the stage for a more indexable and cache-friendly V4 format. Its release cadence remained steady and focused. The project shipped three main versions: 1.8.0 in February, 1.9.0 in April, and 1.10.0 in September.

What shipped in 2025

Iceberg 1.8.0 introduced deletion vectors, default column values, and row-level lineage metadata. These features help engines express updates more efficiently while tracking the origin of each record .

Version 1.9.0 expanded type support. Iceberg now includes a variant type for semi-structured data and geospatial types for geometry-based filtering. The release also added nanosecond timestamps and improved the semantics of equality deletes .

By 1.10.0, the project had added encryption key metadata, cleanup logic for orphaned delete vectors, and full compatibility with Spark 4.0 . Partition statistics were made incremental to reduce overhead in large-scale table planning.

These changes matter. Deletion vectors reduce the cost of updates. Default column values simplify table evolution. Variant support opens the door to querying nested JSON and evolving schemas. Together, these features make Iceberg more expressive and more efficient.

What's coming next

The community has started preparing for Format V4. Key goals include native index support and a formal caching model. The Iceberg dev list also agreed to raise the Java baseline to JDK 17, clearing the way for future performance and security improvements .

Work is also underway to extend the REST catalog spec. This will improve consistency across catalogs like Polaris and make multi-engine deployments behave more predictably.

All of this reflects a clear direction. Iceberg is not only stable, but optimized. It is now equipped to support warehouse workloads with ACID guarantees, even on cloud object storage.

Apache Polaris

Polaris is a new incubating project, but in 2025 it made a fast entrance. Its purpose is simple: act as a shared catalog and governance layer for Iceberg tables across multiple query engines. This includes Spark, Flink, Dremio, Trino, StarRocks, and any system that supports Iceberg's REST catalog protocol.

Why Polaris matters

Today, companies often manage Iceberg tables across multiple engines. Each engine needs a way to authenticate, authorize, and operate on metadata safely. Polaris fills that gap. It provides a consistent API, stores policies centrally, and handles short-term credential vending through built-in integrations with cloud providers.

This makes Polaris one of the first Iceberg-native catalogs to support full multi-engine access, with RBAC and table-level security as first-class features.

What shipped in 2025

Polaris released three versions in its first year: 1.0.0-incubating in July, 1.1.0 in September, and 1.2.0 in October.

The first release included core catalog APIs, a PostgreSQL-backed persistence layer, Quarkus runtime, and initial support for snapshot and compaction policies. It also supported external identity providers, ETag-based caching, and federated metadata views .

Version 1.1.0 added Hive Metastore integration, support for S3-compatible stores like MinIO, and improvements to modularity and CLI tooling .

Version 1.2.0 focused on governance. It expanded RBAC, introduced fine-grained update permissions, and added event logging. AWS Aurora IAM login support also shipped, helping teams standardize credentials across engines .

What's coming next

Polaris is not standing still. Active mailing list discussions show interest in idempotent commit operations, improved retries, and broader NoSQL compatibility. The project is also planning to support Delta Lake tables through its generic table APIs.

Polaris is already production-ready for Iceberg. It supports time travel, commit retries, STS credential vending, and a policy-based governance model. These capabilities make it the metadata backbone of an open lakehouse.

Apache Parquet

Parquet is the disk format most Iceberg tables use. In 2025, the project focused on performance and long-term maintainability. While its interface has changed little, its internals received key upgrades.

What shipped in 2025

The biggest release was Parquet Java 1.16.0 in September. It removed legacy Hadoop 2 support, raised the Java baseline to 11, and enabled vectorized reads by default. These changes help projects like Iceberg, Trino, and Spark take advantage of faster scan paths with less configuration .

The update also refreshed core dependencies like Protobuf and Jackson, fixed bugs in nested field casting, and added CLI support for printing size statistics. For teams managing data layout at scale, this makes table introspection simpler and safer.

On the C++ side, version 12.0 of the Parquet format finalized support for Decimal32 and Decimal64 encodings. These types make aggregations and filters on fixed-point numbers more space-efficient .

What's coming next

The Parquet community has begun discussing what a V3 format might look like. Topics include FSST-based string encoding, cleaner metadata layouts, and faster bloom filter indexing. These ideas aim to reduce scan times and improve filter pushdown without breaking compatibility .

The dev list also revisited lingering V2 features like optional checksums and page-level statistics. There is consensus that these will stabilize in 2026, completing the long tail of V2 work before any format transition.

Parquet’s future is evolutionary, not disruptive. The team is focused on speed, compatibility, and precision. That’s exactly what Iceberg and other engines need from their storage format.

Apache Arrow

Arrow provides the in-memory columnar format that many engines use to exchange data without copying or re-encoding. In 2025, the project extended its feature set, added new bindings, and continued improving compute performance.

What shipped in 2025

Arrow released versions 20.0.0 in April, 21.0.0 in July, and 22.0.0 in October. Each brought changes across the stack, including C++, Python, Java, and R bindings.

The October release expanded compute functions with new regex matchers, selection kernels, and logical operators. It also improved CSV read/write performance, added support for attrs in Pandas DataFrames, and stabilized Decimal32 and Decimal64 support across languages .

Arrow Flight, the RPC layer, shipped a working SQL client implementation. This lays the groundwork for distributed query pushdown using Arrow buffers. Timezone-aware types also advanced, with the community approving a new TimestampWithOffset type to better handle UTC offsets in analytical workflows .

Language support improved too. Arrow released official wheels for modern Linux platforms, added MATLAB bindings, and expanded test coverage for R and Julia. These improvements reduce friction when adopting Arrow across new platforms.

What's coming next

Arrow’s roadmap points toward broader Flight SQL adoption, faster filter and projection kernels, and more alignment between language libraries. Mailing list discussion shows active work on offset encoding, enum types, and compression improvements.

More importantly, Arrow is no longer just a format. It’s becoming an interoperability layer for lakehouse engines. With zero-copy sharing across Spark, Dremio, DuckDB, and beyond, Arrow enables the low-latency experience users expect from a warehouse.

Arrow’s 2025 work reinforced that direction: fast, portable, and deeply integrated with the tools that matter.

Wrapping up

Apache Iceberg, Polaris, Parquet, and Arrow all pushed forward in 2025. Each project focused on practical features that improve performance, governance, or compatibility. Together, they form a foundation for a warehouse experience on open data.

This year’s progress wasn’t about experimentation. It was about consolidation. The features that shipped—from deletion vectors to vectorized reads to Flight SQL—are already in production. They make it easier to build, operate, and scale lakehouse systems.

In 2026, expect the conversation to shift from format maturity to engine convergence. With multi-engine catalogs, index-aware tables, and in-memory interoperability in place, the future looks a lot more accessible. And a lot faster.

dremioframe & iceberg - Pythonic interfaces for Dremio and Apache Iceberg

Alex Merced — Fri, 05 Dec 2025 09:00:00 GMT

Modern data teams want simple tools to work with Iceberg tables and Dremio. Two new Python libraries now make that work easier. The first is DremioFrame. It gives you a clear set of functions for managing your Dremio Cloud or Dremio Software project through code. The second is IceFrame. It gives you a direct way to create and maintain Iceberg tables using PyIceberg and Polars with native extensions. Both libraries are in alpha. This is the best time to try them, share your ideas, and report issues.

You can test them with a free 30-day Dremio Cloud trial that includes $400 in credits. Sign up here to get started . The trial includes a built-in Apache Polaris-based Iceberg catalog (on the ui you'll see a namespaces section, that's the catalog), so you can create tables and explore them from both libraries. This lets you know how the tools fit into real workflows with no setup.

The goal of both libraries is simple. They remove friction. They give you short, readable code. They help you move from idea to result with less effort. They both have built-in AI Agents for assisting you generate code using the library and more. Early feedback from real users will shape their future. Your tests and your questions will guide the next steps. This article introduces the two projects and shows how they work together.

Why These Libraries Exist

Python is the language many teams use for data work. People write scripts, build pipelines, and test ideas in notebooks. Yet working with Iceberg tables or the Dremio REST API often means long code and many repeated steps. These two libraries remove that weight.

DremioFrame gives you a direct way to manage your Dremio catalog, users, views, and jobs. You write clear code that creates folders, defines views, and handles security rules. You no longer need to build each API request by hand.

IceFrame gives you a focused set of tools for Iceberg tables. You can compact files, evolve partitions, and run maintenance tasks with short commands.

Both libraries aim to shorten the path from idea to action. They help you test new patterns, share scripts with your team, and work with Iceberg and Dremio in a direct way.

Meet DremioFrame

DremioFrame is a Python client for Dremio Cloud and Dremio Software. It wraps the REST API in a clean set of methods. You can manage sources, folders, views, tags, and security rules with short commands. You can also run SQL and work with query results as DataFrames.

The library gives you a clear structure. You access the catalog through client.catalog. You manage users and roles through client.admin. You can also manage reflections that speed up queries. Each action is a direct Python call that maps to a known Dremio feature.

The design is simple. You write code that creates a source, builds a view, assigns a policy, or deletes an item. You do not handle request URLs or version tags yourself. This helps teams move faster and keep their scripts readable.

DremioFrame fits well in automation. You can create large batches of folders or datasets through parallel calls. You can also use it in small scripts that update a single view. The goal is to make Dremio easier to use in everyday work.

Meet IceFrame

IceFrame is a Python library that gives you direct control over Iceberg tables. It focuses on clear commands that help you maintain data and keep tables fast. You can compact small files, sort data, evolve partitions, and clear old snapshots. Each task uses a short call that reflects the action you want to take.

The library also supports Iceberg views when the catalog allows it. You can define a view with a simple SQL string and replace it when your logic changes. You can also call stored procedures that handle cleanup and maintenance. This includes rewriting files, removing orphan files, and keeping only recent snapshots.

IceFrame includes an AI assistant for table exploration. You can ask questions in plain language. The tool can show schemas, write example code, and suggest filters or joins. This helps new users learn how the data is shaped and how to work with it.

The goal is steady control with minimal code. You keep your tables healthy and easy to query. You also gain tools to understand your data without long setup or manual checks.

Why Dremio Cloud Is the Best Place to Try Them

Dremio Cloud gives you a smooth way to test both libraries. The trial includes a built-in Iceberg catalog with hosted storage (you can use your own storage with a non-trial account), so you can create tables right away. You do not need to run a separate service or set up extra storage. You write code, create a table, and see it in the Dremio catalog within seconds.

The free 30-day trial includes $400 in credits. You can sign up at the get started page. This gives you enough room to explore IceFrame operations, build views with DremioFrame, and test how the two tools work together.

The setup is light. You create a personal access token, connect through Python, and begin writing code. You can also switch between the console and your scripts to see changes in real time. This makes the trial a strong place for experiments, quick tests, and early feedback.

Using Both Libraries Together

You can use IceFrame and DremioFrame in the same workflow. IceFrame lets you create and shape Iceberg tables locally. DremioFrame lets you see those tables in the catalog, build views on top of them alongside other databases/lakes/warehouses, and apply rules for access or masking. This gives you one flow from data creation to data use.

A simple pattern looks like this. You can write to and manage lightweight Iceberg tables using iceframe for local processing, and use dremioframe to work with Dremio for extensive data processing and query federation with databases/lakes/warehouses, and to curate a semantic and governance layer on top of your data.

You do not move between many tools. You do not manage long request bodies. You write small blocks of code that express the action you need. This helps teams test new ideas and keep their work easy to read and share.

How to Get Started

You can install both libraries with a single step. Run pip install dremioframe and pip install iceframe. You can then import them in any script or notebook. This gives you direct access to the Dremio catalog and your Iceberg tables.

You do not need to clone the repos to use the tools. Cloning is only needed if you want to read the source code or contribute changes. Most users will install the packages from PyPI and begin writing code right away.

After installation, you create a personal access token in Dremio Cloud. You pass that token to DremioFrame when you create the client. You also point IceFrame at your Iceberg catalog. Once this is done, you can create tables, define views, and run cleanup tasks with short commands.

Side-by-Side Examples

The two libraries serve different roles, but they work well together. The examples below show how to connect, run a simple query, and create a table in each library. The code stays short in both cases.

Connect

DremioFrame

from dremioframe.client import DremioClient

client = DremioClient(
    token="YOUR_DREMIO_CLOUD_PAT",
    project_id="YOUR_PROJECT_ID"
)

IceFrame

from iceframe import IceFrame

ice = IceFrame(
    {
        "uri": "https://catalog.dremio.cloud/api/iceberg/v1",
        "token": "YOUR_DREMIO_CLOUD_PAT",
        "project_id": "YOUR_PROJECT_ID"
    }
)

Run a Query

DremioFrame

df = client.sql.run("SELECT 1 AS value")
print(df)

IceFrame

result = ice.query("some_table").limit(10).execute()
print(result)

Create a Table

DremioFrame You create a view or dataset through the catalog. Here is a simple view example.

client.catalog.create_view(
    path=["Samples", "small_view"],
    sql="SELECT * FROM Samples.samples.Employees"
)

IceFrame You create an Iceberg table by writing data.

from datetime import datetime

data = [
    {"id": 1, "name": "Ada", "created_at": datetime.utcnow()},
    {"id": 2, "name": "Max", "created_at": datetime.utcnow()}
]

ice.create_table("my_table", data=data)

These examples show the contrast. DremioFrame works with the Dremio catalog. IceFrame works with Iceberg storage. When used together, they give you a complete path from data creation to query.

Query Builder Examples

Both libraries include a query builder. Each builder keeps the code readable and avoids long SQL strings. The examples below show how each one works.

DremioFrame Query Builder

DremioFrame can build SQL through a fluent API. You call client.table(...) to start. You then add filters, selects, joins, or limits. The builder compiles the final SQL when you run the query.

# Start with a table in the Dremio catalog
df = (
    client.table("Samples.samples.Employees")
        .select("employee_id", "full_name", "department")
        .filter("department = 'Engineering'")
        .limit(5)
        .run()
)

print(df)

This pattern helps when you want to build queries from variables or reuse parts of the logic. The SQL stays clean, and the structure is easy to read.

IceFrame Query Builder

IceFrame includes a builder for Iceberg tables. You call ice.query("table_name") to start. You can then filter rows, pick columns, join tables, or sort results. The builder runs the final plan with execute(). It will determine what parts of the query should be used for Iceberg predicate pushdown and what should be handled after scanning the data for better peformance.

from iceframe.expressions import Column

result = (
    ice.query("my_table")
        .filter(Column("id") > 10)
        .select("id", "name")
        .sort("id")
        .limit(5)
        .execute()
)

print(result)

This pattern keeps the logic simple. You express intent with short steps. The code stays close to how you think about the data.

Both builders help you avoid long SQL strings. They also make it easier to share examples with your team and adapt them to new cases.

Agents and Procedures

Both libraries include features that help you work faster with less manual code. Each tool offers an agent that can guide you through common tasks. IceFrame also includes direct access to Iceberg procedures that keep tables healthy.

Agents

DremioFrame Agent
DremioFrame includes an optional agent that can help you work with DremioFrame and Dremio. It can help you write queries, write DremioFrame scripts, and much more.

IceFrame Agent
IceFrame includes a chat agent for Iceberg tables. You can ask about table schemas, filters, and joins. The agent can write Python code for common IceFrame tasks. It can also explain how to compact files or clean snapshots. This helps new users understand how each feature works. It also helps teams share patterns in a simple way.

IceFrame Procedures

IceFrame gives you access to Iceberg maintenance procedures. These keep data clean and reduce the cost of reading tables. You call each procedure with a short command.

# Rewrite data files
ice.call_procedure("my_table", "rewrite_data_files", target_file_size_mb=256)

# Remove old snapshots
ice.call_procedure("my_table", "expire_snapshots", older_than_ms=7 * 24 * 3600 * 1000)

# Remove orphan files
ice.call_procedure("my_table", "remove_orphan_files")

# Fast-forward a branch
ice.call_procedure("my_table", "fast_forward", to_branch="main")

These steps help you keep tables tidy. They reduce file counts, remove unused data, and keep history at a safe size. You can schedule them or run them by hand. Paired with the agent, you have a clear path from learning a task to running it.

The two libraries share a goal. They help you act faster and with less effort. The agents guide you. The procedures handle the work that keeps your tables stable.

Introducing dremioframe - A Pythonic DataFrame Interface for Dremio

Alex Merced — Sat, 29 Nov 2025 09:00:00 GMT

If you're a data analyst or Python developer who prefers chaining expressive .select() and .mutate() calls over writing raw SQL, you're going to love dremioframe — the unofficial Python DataFrame library for Dremio (currently in Alpha).

Dremio has always made it easy to query across cloud and on-prem datasets using SQL. Some users prefer the ergonomics of DataFrame-style APIs, where transformations are composable, readable, and testable — especially when working in notebooks or building data pipelines in Python.

That’s where dremioframe comes in. It bridges the gap between SQL and Python by letting you build Dremio queries using intuitive DataFrame methods like .select(), .filter(), .mutate(), and more. Under the hood, it still generates SQL and pushes down queries to Dremio, but you write it the way you're used to in Python.

Want to try this yourself?
You can sign up for a free 30-day trial of Dremio Cloud, which includes full access to Agentic AI features, native Apache Iceberg integration, and support for all Iceberg catalogs (e.g. AWS Glue, Nessie, Snowflake, Hive, etc.).
Or if you'd rather run Dremio locally for free, check out the Community Edition setup guide. Community Edition doesn’t include Agentic AI or full catalog support, but still lets you run federated queries and work with some Iceberg catalogs like Glue and Nessie.

In this post, we’ll walk through how to get started with dremioframe—from installing the library and configuring authentication, to writing powerful queries using SQL, DataFrame chaining, and expression builders. We’ll wrap up with a look at some of the more advanced features it unlocks for analytics, ingestion, and administration.

Let’s dive in.

Installing `dremioframe` and Setting Up Your Environment

To get started, you’ll need to install the dremioframe Python package. It’s published on PyPI and can be installed with pip:

pip install dremioframe

Once installed, you’ll need to set up authentication so the library can connect to your Dremio instance. The easiest way to do this is by setting environment variables in a .env file or directly in your shell.

For Dremio Cloud (recommended for full feature access):

In your .env file (or shell), set the following:

DREMIO_PAT=<your_personal_access_token>
DREMIO_PROJECT_ID=<your_project_id>
DREMIO_PROJECT_NAME=<your_project_name>

These credentials can be generated in your Dremio Cloud account by going to project settings.

Don’t have an account?

Start your free 30-day trial of Dremio Cloud to use dremioframe with Agentic AI, native Apache Iceberg support, and full access to all Iceberg catalogs.

For Dremio Community Edition (local setup):

If you're running Dremio locally, for example using the Community Edition, you’ll use a different set of environment variables or pass connection parameters directly in code:

DREMIO_HOSTNAME=localhost
DREMIO_PORT=32010
DREMIO_USERNAME=admin
DREMIO_PASSWORD=password123
DREMIO_TLS=false

Not ready for the cloud yet?

You can try the Community Edition locally by following this guide. It supports federated queries and works with some Iceberg catalogs (like AWS Glue and Nessie), though it doesn’t include the AI features or full catalog support available in Dremio Cloud and Enterprise.

With your environment configured, you’re ready to connect to Dremio and start querying like a Pythonista.

Creating a Dremio Client (Sync or Async)

Once your environment is set up, the next step is to create a DremioClient instance. This object is your entry point for running queries with dremioframe.

Synchronous Client

For most use cases, the synchronous client is sufficient and straightforward to use. If you've set your environment variables, you can initialize the client like this:

from dremioframe.client import DremioClient

client = DremioClient()  # reads config from environment

If you prefer to pass credentials explicitly (useful in scripts or when using the Community Edition), you can do:

client = DremioClient(
    hostname="localhost",
    port=32010,
    username="admin",
    password="password123",
    tls=False  # Set to True if connecting over HTTPS
)

This sets up a connection to your Dremio instance using standard authentication.

Asynchronous Client

If you're working in an async application (e.g., FastAPI, asyncio notebooks, etc.), dremioframe also supports an async client:

from dremioframe.client import AsyncDremioClient

async with AsyncDremioClient(
    pat="YOUR_PAT", 
    project_id="YOUR_PROJECT_ID"
) as client:
    df = await client.table("Samples.samples.dremio.com.zips.json") \
                    .select("city", "state") \
                    .limit(5) \
                    .toPandas()
    print(df)

The async API mirrors the sync one, but allows you to await results in event-driven applications.

Running a Pure SQL Query

Even though dremioframe shines with its DataFrame-style interface, you can still execute raw SQL when needed using the .query() method. This is helpful when you already have a SQL statement or want to run ad hoc queries.

Here’s a simple example that selects city and state from the sample zips dataset:

df = client.query("""
    SELECT city, state
    FROM Samples.samples.dremio.com.zips.json
    WHERE state = 'CA'
    ORDER BY city
    LIMIT 10
""")

print(df)

The result is a lightweight wrapper around a Pandas DataFrame, so you can treat it just like any other DataFrame in Python.

You can also convert it explicitly to a Pandas DataFrame if needed:

pdf = df.toPandas()

Tip: Dremio optimizes and accelerates this query under the hood, especially when you're on Dremio Cloud, where features like autonomous reflection caching are automatic and don't need manual usage.

If you prefer a hybrid approach, dremioframe allows mixing SQL and DataFrame APIs freely—which we'll explore next.

Querying with `.select()` and SQL Functions

The real power of dremioframe comes from its expressive, Pandas-like query builder. You can use .select() to pick columns and include SQL expressions, just like in raw SQL — but with the clarity and structure of method chaining.

Let’s say we want to select a few fields and apply a SQL function like UPPER() to transform the state name:

df = client.table("Samples.samples.dremio.com.zips.json") \
           .select(
               "city", 
               "state", 
               "pop", 
               "UPPER(state) AS state_upper"  # using SQL function
           ) \
           .filter("pop > 100000") \
           .limit(10) \
           .collect()

print(df)

This returns 10 rows where the population is over 100,000 and includes the state_upper column that’s uppercased using Dremio’s SQL engine.

Remember: even though you're using .select(), these expressions are passed through directly to Dremio and fully optimized as part of the SQL query plan.

You can freely combine standard column names with SQL functions, aliases, expressions, and computed columns. This lets you build powerful queries without writing SQL directly.

Want to experiment yourself? Spin up a free Dremio Cloud workspace or try the Community Edition on your laptop.

Transforming Data with `.mutate()`

While .select() is great for choosing and computing columns in one go, .mutate() lets you add new derived columns to an existing selection — much like mutate() in R or .assign() in Pandas.

Let’s take the same query from before and add a new column that calculates population density by dividing population by a fictional land area (just for demo purposes):

df = client.table("Samples.samples.dremio.com.zips.json") \
           .select("city", "state", "pop") \
           .mutate(
               pop_thousands="pop / 1000",               # create a scaled version
               pop_label="CASE WHEN pop > 100000 THEN 'large' ELSE 'small' END"
           ) \
           .filter("state = 'TX'") \
           .limit(10) \
           .collect()

print(df)

In this example:

pop_thousands is a new numeric column.
pop_label is a new string column based on a conditional expression using CASE WHEN.

You can pass any SQL-compatible string expressions into .mutate() using column_name=expression syntax. The expressions are compiled into the underlying SQL query, so performance is fully optimized.

Pro tip: You can chain multiple .mutate() calls if you prefer smaller, incremental steps.

Try experimenting with your own columns! If you’re using Dremio Cloud, you can test these queries on larger datasets with full query acceleration and Iceberg table support. Or run Community Edition locally to follow along with your own data.

Building Queries Programmatically with the Function API

For more complex or dynamic queries, dremioframe provides a powerful function builder API through the F module — similar to how PySpark or dplyr work. This lets you construct expressions programmatically rather than writing raw SQL strings.

Let’s rewrite the previous example using F:

from dremioframe import F

df = client.table("Samples.samples.dremio.com.zips.json") \
           .select(
               F.col("city"),
               F.col("state"),
               F.col("pop"),
               (F.col("pop") / 1000).alias("pop_thousands"),
               F.case()
                 .when(F.col("pop") > 100000, F.lit("large"))
                 .else_(F.lit("small"))
                 .end()
                 .alias("pop_label")
           ) \
           .filter(F.col("state") == F.lit("TX")) \
           .limit(10) \
           .collect()

print(df)

What’s happening here?

F.col("column_name") references a column.
F.case().when(...).else_(...).end() builds a SQL CASE WHEN expression.
F.lit("value") injects a literal value into the expression.
Arithmetic operations like / can be done using Python operators.

This method is especially useful when building queries dynamically — for instance, choosing which fields to include or filter based on user input.

Tip: You can mix function objects with standard strings if needed. Just make sure each expression passed to .select() or .mutate() is either a string or an F object.

Want to try building dynamic queries against Iceberg tables or REST-ingested datasets? Sign up for Dremio Cloud or use Community Edition to test these locally.

What Else Can `dremioframe` Do?

By now, you’ve seen how dremioframe lets you run SQL, build DataFrame-style queries, and programmatically compose logic using expressions. But there’s much more under the hood.

Here’s a quick overview of some additional capabilities you might find useful:

🔄 Joins, Unions, and Time Travel

Join tables with .join(), .left_join(), .right_join(), or .full_join() using either SQL expressions or F functions.
Use .union() to combine rows from two datasets.
Query historical snapshots of Iceberg tables using .at_snapshot("SNAPSHOT_ID").

df = client.table("sales").at_snapshot("123456789")

Iceberg time travel is fully supported in Dremio Cloud and Dremio Enterprise.

Ingest External Data

You can pull data from REST APIs and ingest it directly into Dremio:

client.ingest_api(
    url="https://jsonplaceholder.typicode.com/posts",
    table_name="sandbox.api_posts",
    mode="merge"
)

You can also insert Pandas DataFrames into Dremio tables using:

client.table("sandbox.my_table").insert("sandbox.my_table", data=pd_df)

Analyze, Visualize, and Export

Use .group_by() with aggregates like .sum(), .count(), .mean().

Sort with .order_by(), paginate with .offset(), and chart using .chart().

df.chart(kind="bar", x="state", y="pop")

Export results to local files:

df.to_csv("output.csv")
df.to_parquet("output.parquet")

Data Quality Checks

Built-in expectations let you validate your data:

df.quality.expect_not_null("pop")
df.quality.expect_column_values_to_be_between("pop", min=1, max=1000000)

Admin and Debug Tools

Create and manage reflections (Dremio's Unique Acceleration Layer).
Retrieve and inspect job profiles with .get_job_profile().
Use .explain() to debug SQL plans:

df.explain()

Asynchronous Queries & CLI Access

Use AsyncDremioClient for non-blocking workflows.
Run queries via the command-line tool dremio-cli.

Pro tip: Want to test features like data ingestion, Iceberg catalog browsing, and AI-powered analytics? Dremio Cloud’s 30-day trial gives you full access. For local development, Community Edition is a great way to experiment.

dremioframe is still evolving, but it's already a powerful toolkit for Pythonic analytics on top of Dremio’s lakehouse engine. Whether you're running federated queries, ingesting external APIs, or interacting with Iceberg tables, it helps you stay in the Python world while leveraging all the power of Dremio under the hood.

Conclusion

Whether you're an analyst who loves the clarity of chained DataFrame operations, or a Python developer looking to integrate Dremio into your data pipelines, dremioframe offers a compelling, flexible, and powerful interface to Dremio's lakehouse capabilities.

With just a few lines of code, you can:

Connect securely to Dremio Cloud or Community Edition
Run raw SQL or chain DataFrame-style queries
Add computed columns with .mutate() or build expressions with the F API
Work with federated sources, Apache Iceberg tables, and even ingest external data

By using dremioframe, you get the best of both worlds: the expressiveness of Python and the performance of Dremio’s SQL engine.

Don’t forget — you can sign up for a free 30-day trial of Dremio Cloud to experience all the advanced features like Agentic AI and native support for all Iceberg catalogs.
Or, if you're experimenting locally, try Community Edition to run federated queries and interact with Glue or Nessie-based Iceberg tables.

The dremioframe project is still evolving, but it’s already a powerful toolkit for building readable, maintainable, and scalable data workflows in Python. Give it a try and let us know what you build.

NOTE

dremioframe is an unofficial library and currently in Alpha. Please submit any issues or pull requests to the git repo.

Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)

Alex Merced — Wed, 12 Nov 2025 09:00:00 GMT

Video Playlist of this Walkthough

On November 13, at the Subsurface Lakehouse Conference in New York City, Dremio announced and released Dremio Next Gen Cloud, the most complete and accessible version of its Lakehouse Platform to date. This release advances Dremio’s mission to make data lakehouses easy, fast, and affordable for organizations of any size.

This tutorial offers a hands-on introduction to Dremio and walks through the new free trial experience. With managed storage and no need to connect your own infrastructure or enter a credit card (until you want to), you can explore the full platform, including new AI features, Autonomous Performance Management, and the integrated lakehouse catalog, right away.

What is Dremio?

Dremio is a Data Lakehouse Platform for the AI Era, let's explore what this means.

What is a Data Lakehouse?

A data lakehouse is an architecture that uses your data lake (object storage or Hadoop) as the primary data store for flexibility and openness, then adds two layers to operationalize it like a data warehouse:

A table format such as Apache Iceberg, Delta Lake, Apache Hudi, or Apache Paimon. These formats allow structured datasets stored in Apache Parquet files to be treated as individual tables with ACID guarantees, snapshot isolation, time travel, and more—rather than just a collection of files without these capabilities.
A lakehouse catalog that tracks your lakehouse tables and other assets. It serves as the central access point for data discovery and access control.

Dremio is designed to unify these modular lakehouse components into a seamless experience. Unlike platforms that treat Iceberg as an add-on to proprietary formats, Dremio is built to be natively Iceberg-first—delivering a warehouse-like experience without vendor lock-in.

The Challenges of the Data Lakehouse

While lakehouses offer the benefit of serving as a central source of truth across tools, they come with practical challenges during implementation.

How do you make your lakehouse work alongside other data that isn’t yet in the lakehouse?
How do you optimize storage as data files accumulate and become inefficient after multiple updates and snapshots?
Which catalog should you use, and how do you deploy and maintain it for secure, governed access?

How Dremio’s Platform Supports the Lakehouse

Dremio simplifies many of these challenges with a platform that makes your lakehouse feel like it “just works.” It does this through several powerful features:

Query Federation: Dremio is one of the fastest engines for Apache Iceberg queries, but it also connects to and queries other databases, data lakes, data warehouses, and catalogs efficiently. This means you can start using Dremio with your existing data infrastructure and transition to a full lakehouse setup over time.
Integrated Catalog: Dremio includes a built-in Iceberg catalog, ready to use from day one. This catalog:
- Is based on Apache Polaris, the community-led standard for lakehouse catalogs
- Automatically optimizes Iceberg table storage, eliminating manual tuning
- Provides governance for both Iceberg tables and SQL views with role-based and fine-grained access controls
End-to-End Performance Management: Managing query performance can be time-consuming. Dremio reduces this burden by automatically clustering Iceberg tables and applying multiple layers of caching. One key feature is Autonomous Reflections, which accelerate queries behind the scenes based on actual usage patterns—improving performance before users even notice a problem.
Semantic and Context Layer: Dremio includes a built-in semantic layer where you can define business concepts using SQL views, track lineage, and write documentation. This structure not only supports consistent usage across teams but also provides valuable context to AI systems for more accurate analysis.
AI-Native Features: Dremio Next Gen Cloud includes a built-in AI agent that can run queries, generate documentation, and create visualizations. For external AI systems, the MCP server gives agents access to both data and semantics. New AI functions also let you work with unstructured data for expanded analytical possibilities.

Dremio aims to provide a familiar and easy SQL interface to all your data.

Registering For Dremio Trial

To get started with your Dremio Trial, head over to the Getting Started Page and create a new account with your preferred method.

If using Google/Microsoft/Github you'll be all right up after authenticating, if signing up with your email you'll get an email to confirm your registration.

When you create a new Dremio account, it automatically creates a new Organization, which can contain multiple Projects. The organization will be assigned a default name, which you can change later.

On the next screen, you’ll name your first project. This initial project will use Dremio’s managed storage as the default storage for the lakehouse catalog.

If you prefer to use your own data lake as catalog storage, you can create a new project when you're ready. Currently, only Amazon S3 is supported for custom catalog storage, with additional options coming soon.

Even though S3 is the only supported option for Dremio Catalog storage at the moment, Dremio still allows you to connect to other Iceberg catalogs backed by any cloud storage solution and data lakes using its wide range of source connectors.

Now you'll be on your Dremio Dashboard where you'll wait a few minutes for your organization to be provisioned.

Once the environment is provisioned you'll see several options including a chat box to work with the new integrated Dremio AI Agent which we will revisit later in this tutorial.

One of the best ways to get started is by adding data to Dremio by clicking add data which will open a window where you can either:

Connect an existing Database, Data Lake, Data Warehouse or Data Lakehouse catalog to begin querying data you have in other platforms
Upload a CSV, JSON or PARQUET file which will convert the file into an Iceberg table in Dremio Catalog for you to be able to query.

If looking for some sample files to upload, Kaggle is always a good place to find some datasets to play with.

Although for this tutorial let's use SQL to create tables in Dremio Catalog, insert records into those tables and then query them.

Curating Your Lakehouse

Let's go visit the data explorer to show you how you will navigate your integrated catalog and other datasources. Click on the second icon from the top in the menu that is to the left of the screen that looks like a table, this will take you to the dataset explorer.

In the dataset explorer you'll see two sections:

Namespaces: This is the native Apache Polaris based catalog that belongs to your project. You create namespaces as top-level folders to organize your Apache Iceberg Tables and SQL Views (view SQL can refer to Iceberg and Non-Iceberg datasets, like joining an Iceberg table and a Snowflake Table).
Sources: These are the other sources you've connected to Dremio using Dremio's connectors. You can open up a source and see all the tables available inside of it from the dataset explorer.

Please click on the plus sign next to "namespaces" and add a new namespace called "dremio" this will be necessary to run some SQL scripts I give you without needing to modify them.

Now you'll see the new dremio namespace and in their we can create new tables and views. You may notice there is already a sample data namespace which includes a variety of sample data you can use to experiment with if you want.

Running Some SQL

Now head over to the the "SQL Runner" a full SQL IDE built right into the Dremio experience which includes autocomplete, syntax highlighting, function lookup, the typical IDE shortcuts and much more. It is accessed by clicking the third menu icon which looks like a mini terminal window.

Let me call out a few things to your attention:

On the left you'll notice a column where you can browse available datasets you can use this drag dataset names or column names into your queries so you don't have to type things out everytime.
You'll notice this column has a second tab called scripts, you can save the SQL in any tab as a script you can comeback to later, great for template scripts or scripts you haven't finished yet.
The SQL editor is on top and the results viewer is on the bottom, if you run multiple SQL statements the results viewer will give you a tab for the result of each query run making it easy to isolate the results of different parts of your script.

There is a lot more to learn about the SQL runner, but let's go ahead and run some SQL. I've written several SQL scripts you can copy into the SQL Runner and run as is. Choose any of the below and copy them into SQL runner and run the SQL. Give the code a look over, comments in the code help explain what it is doing.

The SQL for the majority of these examples follow a similar pattern:

Create sub folder for the example
Create a bronze/silver/gold subfolder in that subfolder
Create and insert data in the base tables in the raw folder
Join and model using a SQL view to create the silver layer
Create a use case specific views from the silver view in the gold layer

This represents a very typical way of using Dremio where you model your datasets not by replicating your data but logically with SQL views. Dremio's autonomous reflections feature will see how these views are queried and dynamically determine what views should be materialized into the Dremio's reflection cache without anyone having to lift a finger keeping everything performance on storage and compute usage. A data engineer could also manually trigger the creation of a reflection and Dremio will assign that reflection a score to help understand whether the reflection is providing value or not, we'll show this when we got over Dremio's settings UI.

Two of the examples do something a little different:

The Manufacturing example uses SQL views to create a medallion architecture, but it then also creates health check views which allow you to get a quick glance at the health of a dataset by aggregating nulls, duplicates, etc.
The retail example shows you what it may look like if you don't want to model your data logically but physically using CTAS (Create Table As) statements to materialize the data as physical tables through the bronze/silver/gold layers.

AI Visualizations

Now that you've added some data to your lakehouse but running some of my example scripts, let's show you some of the benefits of Dremio's new integrated AI Agent.

While if you are planning to build long lived BI dashboards you should still leverage integrations most BI tools like Tableau and Power BI have with Dremio, but if you need to just seem something visual quick or just need something to screenshot into a presentation the AI Agent can be quite helpful.

So let's say I run the supply chain example, I could prompt the AI Agent with a prompt like:

Can you create a bar chart based on the data in the gold layer of my supply chain data

The AI Agent will pull the list of tables your particular user has access to and see if it finds the dataset I loosely referred to. It will then pull the schema and other data from the semantic layer to better understand the dataset before deciding how to best visualize it. But of course my prompt was vague and the AI agent decides to confirm some details with me:

I can create that bar chart — quick clarifying questions before I run the query
I inspected your gold layer and found the view dremio.supply_chain.gold.daily_supplier_delivery_kpis with these columns:

supplier_id (string)
receipt_date (date)
lines_received (int)
otif_rate (double)
on_time_rate (double)
in_full_rate (double)
total_received_qty (int)
total_order_qty (int)
gross_fill_rate (decimal)
To produce the bar chart I need to know which metric and timeframe you want visualized. Common options:

Average OTIF per supplier (bar per supplier showing average “otif_rate” across the dataset)
Average gross fill rate per supplier (bar per supplier showing average “gross_fill_rate”)
Total lines received per supplier for a specific date or date range (sum of “lines_received”)
Side-by-side bars per supplier showing two metrics (e.g., OTIF and gross_fill_rate) — I can produce grouped bars if you confirm.
Suggested defaults (if you don’t specify):

Metric: avg(otif_rate)
Timeframe: all available dates
Chart: vertical bar, supplier on x-axis, metric on y-axis, ordered by metric desc
If that works, I will run this SQL against the gold view and build the bar chart:

SELECT “supplier_id”, AVG(“otif_rate”) AS “avg_otif_rate”, AVG(“gross_fill_rate”) AS “avg_gross_fill_rate” FROM “first-project”.“dremio”.“supply_chain”.“gold”.“daily_supplier_delivery_kpis” — optionally add: WHERE “receipt_date” BETWEEN DATE ‘2025-08-01’ AND DATE ‘2025-08-31’ GROUP BY “supplier_id” ORDER BY “avg_otif_rate” DESC

Please confirm:

Which metric? (otif_rate, gross_fill_rate, lines_received, total_received_qty, etc.)
Timeframe (all data, a date, or a date range). If you say a month/week/day, I will assume the current year unless you specify otherwise.
Chart type (single bar, grouped bars for two metrics).
After you confirm I will run the query and produce the bar chart for you.

I respond with

1. Average OTIF per supplier
2. All data
3. Vertical Bar

It then explains to the me it's thought process to illustrate where it needs more detail, I notice at this point it is including the project name first-project in the table name which is not necessary as this is assumed so it should start with the namespace dremio. I tell it this and get back my visualization.

We found success starting with quite a vague question but working with the AI we were able to get a visualization of a useful metric within a few minutes.

AI Function

Using your data to create visualization isn't the only cool AI integration in the Dremio Arsenal. Dremio also has added a variety of new SQL AI Functions which allow you to do a variety of things like turn unstructured data into structured data. Let's see a very simple example you can run right in your SQL runner assuming you have a dremio namespace.

-- Create the recipes table with an ARRAY column for ingredients (sample rows)
-- Note: this uses CREATE TABLE AS SELECT to create a physical table with sample data.
CREATE FOLDER IF NOT EXISTS dremio.recipes;
CREATE TABLE IF NOT EXISTS dremio.recipes.recipes AS
SELECT 1 AS "id",
       'Mild Salsa' AS "name",
       ARRAY['tomato','onion','cilantro','jalapeno','lime'] AS "ingredients",
       CURRENT_TIMESTAMP AS "created_at"
UNION ALL
SELECT 2, 'Medium Chili', ARRAY['beef','tomato','onion','chili powder','cumin','jalapeno'], CURRENT_TIMESTAMP
UNION ALL
SELECT 3, 'Spicy Vindaloo', ARRAY['chicken','chili','ginger','garlic','vinegar','habanero'], CURRENT_TIMESTAMP;

-- Create View where AI is used to classify each recipe as Mild, Medium or Spicy
CREATE OR REPLACE VIEW dremio.recipes.recipes_enhanced AS SELECT id,
       name,
       ingredients,
       AI_CLASSIFY('Identify the Spice Level:' || ARRAY_TO_STRING(ingredients, ','), ARRAY [ 'mild', 'medium', 'spicy' ]) AS spice_level
from   dremio.recipes.recipes;

The First SQL statement creates a table of recipes where the ingredients are an array of strings
The Second SQL statement create a view where we use the AI_CLASSIFY function to prompt the AI given the ingredients whether the recipe is mild, medium or spicy.

With these AI functions you can also use it pull data from JSON files or folders of images to generate structured datasets. Imagine taking a folder of scans of paper applications and turning them into an iceberg table with all the right fields by having the AI scan these images, this is kind of use case made possible by these functions.

Dremio Jobs Pane

Want to see what queries are coming or investigate deeper why a query may have failed or taken longer than expecting, the Dremio job pane which is the next option on the left menu will allow you see all your jobs and then click on them to see exaustive detail on how they were processed.

Dremio Settings

If you click on the last menu item, the gear, you'll get two options:

Project Settings
Org Settings

Project Settings

You can find project info like:
- project name and id (project names are fixed, org names can change)
- MCP server url to connect your external AI agent to leverage your Dremio instance
- JDBC url to connect to Dremio using external JDBC clients and in custom scripts

NOTE: SQL can be sent to Dremio for execution outside of Dremio's UI using JDBC, ODBC, Apache Arrow Flight and Dremio's REST API. Refer to docs.dremio.com for documentation on how to leverage these interfaces.

Also in project settings you'll find sections like:

Catalog: To update catalog settings like how often metadata refresh should happen. (Note: Lineage view is based on the metadata as of the last refresh so if something isn't reflected in lineage the metadata may not have refreshed yet. You can either make the metadata refresh more frequently or wait till it refreshes on its current schedule.)
Engines: For managing your different Dremio execution engines, what is their size, when they should spin up and when they should spin down
BI Tools: Enable or disable Tableu and Power BI buttons
Monitor: Dashboard to monitor Dremio project health
Reflections: See scores on reflections you have created, you can also delete reflections if you no longer need them from here
Engine Routing: Create rules for which jobs should go to which engines, for example jobs from certain users may be routed to their own engine which is tracked for charge backs.
Preferences: Turn on and off certain Dremio features

Organization Settings

Under Organizations settings you'll find:

name of org, which can be changed
Manage authentication protocols
Manage projects, users, and roles

User Settings

At the very bottom left corner there is a button to see settings for the individual user. The Main use for this is to change to dark/light more and to create PAT tokens for authenticating external clients.

Granting Access

Once you create new non-admin users in your Dremio org, they'll have zero access to anything so you'll need to give them precise access to particular projects, namespaces, folders, sources etc.

While you can do this for an individual user, it will likely be easier to create "roles" you can grant access to groups of users with. Below is the example of the kind of SQL you may use to grant access to a single namespace for a new user.

-- Give Permissions to project
GRANT SELECT, VIEW REFLECTION, VIEW JOB HISTORY, USAGE, MONITOR,
       CREATE TABLE, INSERT, UPDATE, DELETE, DROP, ALTER, EXTERNAL QUERY, ALTER REFLECTION, OPERATE
ON PROJECT
TO USER "alphatest2user@alexmerced.com";

-- Give Permissions to Namespace in Catalog
GRANT ALTER, USAGE, SELECT, WRITE, DROP on FOLDER "dremio" to USER "alphatest2user@alexmerced.com";

-- Give Permissions to a Folder in the namespace
GRANT ALTER, USAGE, SELECT, WRITE, DROP on FOLDER dremio.recipes to USER "alphatest2user@alexmerced.com";

Connecting your Dremio Catalog to Other Engines Like Spark

Now you can connect to the Dremio Platform using JDBC/ODBC/ADBC-Flight/REST and send SQL to Dremio for Dremio to execute which I hope you take full advantage of. Although, sometimes you are sharing a dataset in your catalog with someone else who wants to use their preferred compute tool. Dremio Catalog bein Apache Polaris based supports the Apache Iceberg REST Catalog SPEC meaning it can connect to pretty much to any Apache Iceberg supporting tool. Below is an example of how you'd connect in Spark.

Run a local spark envrionment using the following command:

docker run -p 8888:8888 -e DREMIO_PAT={YOUR PAT TOKEN} alexmerced/spark35nb:latest

Then use the following code to run spark code against Dremio Catalog (keep in mind the CATALOG_NAME variable should match your project name).

import os
import pyspark
from pyspark.sql import SparkSession
from pyspark.sql import Row

# Fetch Dremio base URL and PAT from environment variables
DREMIO_CATALOG_URI = "https://catalog.dremio.cloud/api/iceberg"
DREMIO_AUTH_URI = "https://login.dremio.cloud/oauth/token"
DREMIO_PAT = os.environ.get('DREMIO_PAT')
CATALOG_NAME = "first-project" # should be project name

if not DREMIO_CATALOG_URI or not CATALOG_NAME or not DREMIO_AUTH_URI or not DREMIO_PAT:
    raise ValueError("Please set environment variables DREMIO_CATALOG_URI, DREMIO_AUTH_URI and DREMIO_PAT.")

# Configure Spark session with Iceberg and Dremio catalog settings
conf = (
    pyspark.SparkConf()
        .setAppName('DremioIcebergSparkApp')
        # Required external packages For FILEIO (org.apache.iceberg:iceberg-azure-bundle:1.9.2, org.apache.iceberg:iceberg-aws-bundle:1.9.2, org.apache.iceberg:iceberg-azure-bundle:1.9.2, org.apache.iceberg:iceberg-gcp-bundle:1.9.2)
        .set('spark.jars.packages', 'org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.2,com.dremio.iceberg.authmgr:authmgr-oauth2-runtime:0.0.5,org.apache.iceberg:iceberg-aws-bundle:1.9.2')
        # Enable Iceberg Spark extensions
        .set('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions')
        # Define Dremio catalog configuration using RESTCatalog
        .set('spark.sql.catalog.dremio', 'org.apache.iceberg.spark.SparkCatalog')
        .set('spark.sql.catalog.dremio.catalog-impl', 'org.apache.iceberg.rest.RESTCatalog')
        .set('spark.sql.catalog.dremio.uri', DREMIO_CATALOG_URI)
        .set('spark.sql.catalog.dremio.warehouse', CATALOG_NAME)  # Not used but required by Spark
        .set('spark.sql.catalog.dremio.cache-enabled', 'false')
        .set('spark.sql.catalog.dremio.header.X-Iceberg-Access-Delegation', 'vended-credentials')
        # Configure OAuth2 authentication using PAT
        .set('spark.sql.catalog.dremio.rest.auth.type', 'com.dremio.iceberg.authmgr.oauth2.OAuth2Manager')
        .set('spark.sql.catalog.dremio.rest.auth.oauth2.token-endpoint', DREMIO_AUTH_URI)
        .set('spark.sql.catalog.dremio.rest.auth.oauth2.grant-type', 'token_exchange')
        .set('spark.sql.catalog.dremio.rest.auth.oauth2.client-id', 'dremio')
        .set('spark.sql.catalog.dremio.rest.auth.oauth2.scope', 'dremio.all')
        .set('spark.sql.catalog.dremio.rest.auth.oauth2.token-exchange.subject-token', DREMIO_PAT)
        .set('spark.sql.catalog.dremio.rest.auth.oauth2.token-exchange.subject-token-type', 'urn:ietf:params:oauth:token-type:dremio:personal-access-token')
)

# Initialize Spark session
spark = SparkSession.builder.config(conf=conf).getOrCreate()
print("✅ Spark session connected to Dremio Catalog.")

# Step 1: Create a namespace (schema) in the Dremio catalog
spark.sql("CREATE NAMESPACE IF NOT EXISTS dremio.db")
# spark.sql("CREATE NAMESPACE IF NOT EXISTS dremio.db.test1")
print("✅ Namespaces Created")

# Step 2: Create sample Iceberg tables in the Dremio catalog
spark.sql("""
CREATE TABLE IF NOT EXISTS dremio.db.customers (
    id INT,
    name STRING,
    email STRING
)
USING iceberg
""")

spark.sql("""
CREATE TABLE IF NOT EXISTS dremio.db.orders (
    order_id INT,
    customer_id INT,
    amount DOUBLE
)
USING iceberg
""")

print("✅ Tables Created")

# Step 3: Insert sample data into the tables
customers_data = [
    Row(id=1, name="Alice", email="alice@example.com"),
    Row(id=2, name="Bob", email="bob@example.com")
]

orders_data = [
    Row(order_id=101, customer_id=1, amount=250.50),
    Row(order_id=102, customer_id=2, amount=99.99)
]

print("✅ Dataframes Generated")

customers_df = spark.createDataFrame(customers_data)
orders_df = spark.createDataFrame(orders_data)

customers_df.writeTo("dremio.db.customers").append()
orders_df.writeTo("dremio.db.orders").append()

print("✅ Tables created and sample data inserted.")

Conclusion

Dremio Next Gen Cloud represents a major leap forward in making the data lakehouse experience seamless, powerful, and accessible. Whether you're just beginning your lakehouse journey or modernizing a complex data environment, Dremio gives you the tools to work faster and smarter—with native Apache Iceberg support, AI-powered features, and a fully integrated catalog.

From federated queries across diverse sources to autonomous performance tuning, Dremio abstracts away the operational headaches so you can focus on delivering insights. And with built-in AI capabilities, you're not just managing data—you’re unlocking its full potential.

If you haven’t already, sign up for your free trial and start building your lakehouse—no infrastructure or credit card required.

The next generation of analytics is here. Time to explore what’s possible.

2025-2026 Guide to Learning about Apache Iceberg, Data Lakehouse & Agentic AI

Alex Merced — Thu, 23 Oct 2025 09:00:00 GMT

The data world is evolving fast. Just a few years ago, building a modern analytics stack meant stitching together tools, ETL pipelines, and compromises. Today, open standards like Apache Iceberg, modular architectures like the data lakehouse, and emerging patterns like Agentic AI are reshaping how teams store, manage, and use data.

But with all this innovation comes one challenge: where do you start?

This guide was created to answer that question. Whether you're a data engineer exploring the Iceberg table format, an architect building a lakehouse, or a developer curious about AI agents that interact with real-time data, this resource will walk you through it. No hype. No fluff. Just a curated directory of the best learning paths, tools, and concepts to help you build a practical foundation.

We will break down the links into many categories to help you find what you are looking for. There is more content beyond what I have listed here and here are two good directories where you can find more content to explore.

Lakehouse Blog Directories We will break down the links into many categories to help you find what you are looking for. There is more content beyond what I have listed here and here are two good directories where you can find more content to explore.

Get Data Lakehouse Books: I've had the honor of getting to participate in some long form written content around the lakehouse, make of these you can get for free, links below:

Lakehouse Community: Below are some links where you can network with other lakehouse enthusiasts and discover lakehouse conferences and meetups near you!

The Data Lakehouse

The idea behind a data lakehouse is simple: keep the flexibility of a data lake, add the performance and structure of a warehouse, and make it all accessible from one place. But turning that idea into a working architecture takes more than just buzzwords. In this section, you'll find tutorials, architectural guides, and practical walkthroughs that explain how lakehouses work, when they make sense, and how to get started, whether you’re running everything on object storage or looking to unify data access across teams and tools.

Apache Iceberg

Apache Iceberg is the table format that makes data lakehouses actually work. It brings support for ACID transactions, schema evolution, time travel, and scalable performance to your cloud storage, without locking you into a vendor or engine. If you’ve ever wrestled with Hive tables or brittle partitioning logic, this section is for you. Here, you'll find beginner-friendly resources, deep dives into metadata and catalogs, and hands-on guides for working with Iceberg using engines like Spark, Flink, and Dremio.

What are Lakehouse Open Table Formats

Table formats are the backbone of the modern lakehouse. They define how data files are organized, versioned, and transacted, bringing warehouse‑level reliability to open storage. This section explores what makes formats like Apache Iceberg, Delta Lake, and Apache Hudi so important. You’ll learn how they handle schema evolution, partitioning, and ACID transactions while staying engine‑agnostic, ensuring your data remains open, performant, and ready for any workload.

Apache Iceberg Tutorials

Getting hands‑on is the fastest way to learn Apache Iceberg. In these tutorials, you’ll spin up local environments, run your first SQL commands, and connect Iceberg tables with catalogs like Apache Polaris or engines like Spark and Dremio. Each guide walks you through setup, basic operations, and troubleshooting so you can move from theory to practice without friction.

Iceberg Migration Tooling and Ingestion

Moving existing datasets into Apache Iceberg doesn’t have to be painful. This section highlights migration patterns, ingestion tools, and automation workflows that make it easier to adopt Iceberg at scale. You’ll find step‑by‑step resources covering snapshot‑based migrations, bulk ingests, and hybrid models that help teams modernize data lakes while minimizing downtime and duplication.

Iceberg Catalogs

A table format is only as useful as the catalog that organizes it. Iceberg catalogs manage metadata, access control, and engine interoperability, essential pieces of a production lakehouse. In this section, you’ll explore the expanding catalog ecosystem, from open implementations like Apache Polaris to commercial and hybrid options. These resources explain how catalogs enable discoverability, governance, and smooth multi‑engine coordination across your data environment.

Apache Iceberg Table Optimization

Keeping Iceberg tables fast requires more than good schema design. Over time, data fragmentation, small files, and metadata sprawl can slow queries and inflate costs. The articles in this section show how to maintain healthy tables through compaction, clustering, and automatic optimization. You’ll also learn how modern platforms like Dremio manage this maintenance autonomously so performance tuning doesn’t become a full‑time job.

Iceberg Technical Deep Dives

Once you understand the basics, the real fun begins. These deep dives unpack how Iceberg works under the hood, covering metadata structures, query caching, authentication, and advanced performance topics. Whether you’re benchmarking, extending the format, or building your own catalog integration, this section will help you understand Iceberg’s architecture and internal mechanics in detail.

The Future of Apache Iceberg

Apache Iceberg continues to evolve alongside emerging workloads like Agentic AI and next‑generation file formats. This section looks ahead at what’s coming; new format versions, engine integrations, and evolving standards such as Polaris and REST catalogs. If you want to stay informed on where Iceberg is heading and how it fits into the broader open‑data movement, start here.

Agentic AI

Agentic AI is a new class of systems that don’t just answer questions, they take action. These agents make decisions, follow workflows, and learn from outcomes, but they’re only as smart as the data they can access. That’s where open lakehouse architectures come in. This section explores the intersection of data architecture and autonomous systems, with content focused on how to power agents using structured, governed, and real-time data from your Iceberg-based lakehouse. From semantic layers to zero-ETL federation, you'll see what it takes to build AI that isn't just reactive, but genuinely useful.

An Exploration of the Commercial Iceberg Catalog Ecosystem

Alex Merced — Tue, 21 Oct 2025 09:00:00 GMT

Get Data Lakehouse Books:

Lakehouse Community:

Apache Iceberg has quickly become the table format of choice for building open, flexible, and high-performance data lakehouses. It solves long-standing issues around schema evolution, ACID transactions, and engine interoperability. Enabling a shared, governed data layer across diverse compute environments.

But while the table format itself is open and standardized, the catalog layer, the system responsible for tracking and exposing table metadata, is where key decisions begin to shape your architecture. How your organization selects and manages an Iceberg catalog can influence everything from query performance to write flexibility to vendor lock-in risk.

This blog explores the current landscape of commercial Iceberg catalogs, focusing on the emerging Iceberg REST Catalog (IRC) standard and how different vendors interpret and implement it. We’ll examine where catalogs prioritize cross-engine interoperability, where they embed proprietary optimization features, and how organizations can approach these trade-offs strategically.

You’ll also learn what options exist when native optimizations aren’t available, including how to design your own or consider a catalog-neutral optimization tool like Ryft.io (when using cloud object storage).

By the end, you'll have a clear view of the commercial ecosystem, and a framework to help you choose a path that fits your technical goals while minimizing operational friction.

The Role of Iceberg REST Catalogs in the Modern Lakehouse

At the heart of every Apache Iceberg deployment is a catalog. It’s more than just a registry of tables, it’s the control plane for transactions, schema changes, and metadata access. And thanks to the Apache Iceberg REST Catalog (IRC) specification, catalogs no longer need to be tightly coupled to any single engine.

The IRC defines a standardized, HTTP-based API that lets query engines like Spark, Trino, Flink, Dremio, and others communicate with a catalog in a consistent way. That means developers can write data from one engine and read it from another, without worrying about format mismatches or metadata drift.

This decoupling brings three major benefits:

Multi-language support: Since the interface is language-agnostic, you can interact with the catalog from tools written in Java, Python, Rust, or Go.
Compute independence: Query and write operations don’t require the catalog to be embedded in the engine, everything runs through REST.

Adoption of the IRC spec is growing rapidly. Vendors like Dremio, Snowflake, Google, and Databricks now offer catalogs that expose some or all of the REST API. This trend signals a broader shift toward open metadata services, where engine choice is driven by workload needs, not infrastructure constraints.

But as we’ll see next, implementing the REST API is only part of the story. The real architectural decisions start when you consider how these catalogs handle optimization, write access, and cross-engine consistency.

Key Considerations When Choosing a Catalog

Picking a catalog shapes how your Iceberg lakehouse runs. The decision affects who can read and write data, how tables stay performant, and how easy it is to run multiple engines. Focus on facts. Match catalog capabilities to your operational needs.

Read-write interoperability.
Some catalogs expose the full Iceberg REST Catalog APIs so any compatible engine can read and write tables. Other offerings restrict external writes or recommend using specific engines for writes. These differences change how you design ingestion and cross-engine workflows.

Server-side performance features.
Catalogs vary in how much they manage table health for you. A few provide automated compaction, delete-file handling, and lifecycle management. Others leave those tasks to your teams and to open-source engines. If you want fewer operational jobs, prioritize a catalog with built-in performance management.

Vendor neutrality versus added convenience.
A catalog that automates maintenance reduces day-to-day work. It also increases dependency on that vendor’s maintenance model. If your priority is full independence across engines then you may prefer a catalog that implements the Iceberg REST spec faithfully so you can plan for external maintenance processes.

Costs and Compatibility Some catalogs may be limited on which storage providers they can work with or may charge just for usage of the catalog even if you use external compute and this should be considered.

A short checklist to evaluate a candidate catalog

Does it implement the Iceberg REST Catalog APIs for both reads and writes?
Does it provide automatic table maintenance or only catalog services?
What write restrictions or safety guards exist for external engines?
Which clouds and storage systems does it support?
Are there extra costs to using the catalog?

Use this checklist when you compare offerings. It helps reveal trade-offs between operational simplicity and multi-engine freedom.

Catalog Optimization: Native vs. Neutral Approaches

Once your Iceberg tables are in place, keeping them fast and cost-effective becomes a daily concern. File sizes grow unevenly, delete files stack up, and query times creep higher. This is where table optimization comes in—and where catalog differences start to matter.

Most commercial catalogs fall into two categories:

Native Optimization Available
Manual Optimization Required

Native Optimization Available

Vendors like Dremio, AWS Glue, and Databricks Unity Catalog offer built-in optimization features that automatically manage compaction, delete file cleanup, and snapshot pruning. These features are often tightly integrated into their orchestration layers or compute engines.

Benefits:

No need to schedule Spark or Flink jobs manually
Optimizations are triggered based on metadata activity
Helps reduce cloud storage costs and improve query performance

Tradeoff:

These features are often proprietary and non-transferable. If you move catalogs or engines, you may lose automation and need to build optimization pipelines.

Catalog-Neutral or Manual Optimization

Some catalogs, including open-source options like Apache Polaris, don't come with built-in optimization. Instead, you have two options:

Run your own compaction pipelines using engines like Spark or Flink. Can also manually orchestrate Dremio's OPTIMIZE and VACUUM commands with any catalog.
Use a catalog-neutral optimization service like Ryft.io, which works across any REST-compatible catalog, but currently only supports storage on AWS, Azure, or GCP. There is also the open source Apache Amoro which automates the use of Spark based optimizations.

This route offers maximum flexibility but requires:

Engineering effort to configure and monitor compaction
Knowledge of best practices for tuning optimization jobs
A way to coordinate across engines to avoid conflicting writes

In short: if optimization is a feature you want off your plate, look for a catalog that handles it natively. If you prefer full control or need a more cloud-agnostic setup, neutral optimization tools or open workflows may serve you better.

What If Native Optimization Doesn’t Exist?

Not every catalog includes built-in optimization. If you're using a minimal catalog, or one that prioritizes openness over orchestration, you’ll need to handle performance tuning another way. That’s not a dealbreaker, but it does require a decision.

Here are the two main paths forward when native optimization isn’t part of the package:

Option 1: Build Your Own Optimization Pipelines

Apache Iceberg is fully compatible with open engines like Apache Spark, Flink, and Dremio. Each of these supports table maintenance features such as:

File compaction
Manifest rewriting
Snapshot expiration

You can schedule these jobs using tools like Airflow or dbt, or embed them directly into your data ingestion flows. This approach works in any environment, including on-prem, hybrid, and cloud.

Pros:

Complete flexibility in how and when you optimize
Can tailor jobs to match data patterns and storage costs
Fully open and vendor-independent

Cons:

Requires engineering effort to build, monitor, and tune jobs
No centralized UI or automation unless you build one

Option 2: Use a Catalog-Neutral Optimization Vendor

Vendors like Ryft.io offer managed optimization services designed specifically for Iceberg. These tools run outside your query engines and handle compaction, cleanup, and layout improvements without relying on any one catalog or engine.

NOTE: Apache Amoro offers an open source optimization tool if looking for an open source option.

Key detail: Ryft currently only supports deployments that store data in AWS S3, Azure Data Lake, or Google Cloud Storage. If you're using on-prem HDFS or other object stores, this may not be viable.

Pros:

No need to manage optimization logic
Works across multiple compute engines and catalogs
Keeps optimization decoupled from platform lock-in

Cons:

Limited to major cloud object storage unless using Apache Amoro
Adds another vendor and billing model to your stack

When native optimization isn’t available, the best path depends on your team’s appetite for operational work. DIY gives you control. Neutral services give you speed. Either way, optimization remains a critical layer—whether you manage it yourself or let someone else handle it.

5. The Interoperability Spectrum

One of the key promises of Apache Iceberg is engine interoperability. The Iceberg REST Catalog API was designed so any compliant engine—whether it's Spark, Flink, Trino, or Dremio—can access tables the same way. But in practice, not all catalogs offer equal levels of interoperability.

Some catalogs expose full read/write access to external engines using the REST API. Others allow only reads—or place restrictions on how writes must be performed. This creates a spectrum, where catalogs differ in how open or engine-specific they are.

Here’s how several major catalogs compare:

Catalog	External Read Access	External Write Access	REST Spec Coverage	Notes
Dremio Catalog	✅ Full	✅ Full	✅ Full	Based on Apache Polaris; full multi-engine support; no cost for external reads/writes
Apache Polaris (Open Source)	✅ Full	✅ Full	✅ Full	Vendor-neutral, open REST catalog, deploy yourself or get managed by Dremio or Snowflake
Databricks Unity Catalog	✅ Full	✅ Full	✅ Full	Optimization services are primarily Delta Lake Centered
AWS Glue & AWS S3 Tables	✅ Full	✅ Full	✅ Full
Google BigLake Metastore	✅ Full	✅ Full	✅ Full (preview)
Snowflake Open Catalog	✅ Full	✅ Full	✅ Full	Based on Apache Polaris; Charged for requests to catalog from external reads/writes
Snowflake Managed Tables	✅ Full	❌ None	❌ None	Tables can be externally read using Snowflake's SDK
Microsoft OneLake	✅ Full (Preview)	✅ Virtualized Writes	✅ Full (preview)	✅ Virtualized via XTable
MinIO AIStor	✅ Full	✅ Full	✅ Full	⚠️ Storage‑level Optimization Only
Confluent TableFlow	✅ Full	⚠️ Limited	✅ Full	⚠️ Fixed Snapshot Retention
DataHub Iceberg Catalog	✅ Full	✅ Full	✅ Full	S3 Only

What This Means for You

If your architecture depends on multiple engines, the safest route is to choose a catalog that:

Implements the full Iceberg REST spec
Allows both reads and writes from all compliant engines
Avoids redirecting writes through proprietary services or SDKs

This isn’t just about standards, it’s about reducing long-term friction. The more interoperable your catalog, the easier it is to plug in new tools, migrate workloads, or share datasets across teams without rewriting pipelines or triggering lock-in.

Architectural Patterns: Choosing the Right Iceberg Catalog for Your Stack

With a clear understanding of feature capabilities across commercial Iceberg catalogs, the next consideration is architectural alignment. How should teams select a catalog based on their engine stack, deployment model, and optimization philosophy?

Here, we explore common deployment patterns and their implications:

Single-Engine Simplicity

Use case: Organizations standardized on one compute engine seeking high performance and low operational overhead.

✅ Benefits:
- Seamless integration between compute and catalog.
- Native optimization features (e.g., OPTIMIZE TABLE, Z-Ordering).
- Simplified access control and performance tuning.
⚠️ Trade-offs:
- May impose file format restrictions (e.g., Parquet-only).
- Optimization tightly coupled to engine but if REST-Spec is ahered to you can still develop your own optimization pipelines.

Recommended Platforms: Dremio, Databricks Unity Catalog, AWS Glue (with managed compute).

Multi-Engine Interop (Spark + Trino + Flink)

Use case: Organizations with complex, multi-engine environments that require consistent metadata across tools and Clouds.

Benefits:
- Use the right engine for the right job (ETL, BI, ML).
- Maximize transactional openness via full IRC support.
Trade-offs:
- Optimization is either manual or vendor-dependent.
- Catalog-neutral solutions may lack server-side performance tuning.

Recommended Platforms: Dremio Enterprise Catalog, Snowflake Open Catalog. (Both based on Apache Polaris)

Streaming-First Architectures

Use case: Teams integrating real-time data from Kafka into the lakehouse for analytics or ML.

Benefits:
- Stream-native catalog (e.g., Confluent TableFlow) materializes Kafka topics into Iceberg tables.
- Seamless schema registration and time-travel.
Trade-offs:
- No schema evolution.
- Limited optimization control (rigid snapshot retention).
- Often designed for read-heavy use cases.

Recommended Platforms: Confluent TableFlow, integrated with external catalogs for downstream processing.

Cloud-Embedded Storage Catalogs

Use case: Teams deploying AI or analytics workloads in private/hybrid cloud environments.

Benefits:
- Built-in REST Catalog support directly within storage (MinIO AIStor).
- Simplifies deployment—no separate metadata layer deployment.
- High concurrency and transactional consistency at scale.
Trade-offs:
- Tightly bound to object storage vendor.
- No native table optimization

Recommended Platforms: MinIO AIStor (on-premise/private cloud), AWS S3 Tables (cloud-native equivalent).

Governance-Led Architectures

Use case: Enterprises prioritizing metadata lineage, compliance, and discovery.

Benefits:
- Centralized metadata layer for observability and access management.
- Easy discovery and tracking across teams and tools.
Trade-offs:
- No native write capabilities (metadata-only catalog).
- Optimization must be handled by external systems.

Recommended Platforms: DataHub Iceberg Catalog (OSS or Cloud) or using an external catalog (Dremio Catalog, Apache Polaris) connected into Datahub.

Each pattern has architectural trade-offs. Rather than seeking a perfect catalog, successful teams prioritize alignment with workflow needs: engine independence, optimization automation, governance, or real-time ingestion. In some cases, hybrid strategies, like dual catalogs or catalog, neutral optimization overlays—provide the best of both worlds.

Optimization Strategy Trade-offs: Native, Manual, or Vendor-Neutral

Once an organization selects a catalog, the next major architectural decision is how to maintain and optimize Iceberg tables. While the IRC standard guarantees transactional consistency, it says nothing about how tables should be optimized over time to preserve performance and control storage costs.

Three primary approaches emerge:

1. Native Optimization (Catalog-Integrated Automation)

Many commercial catalogs offer built-in optimization features tightly coupled with their own compute engines. These include operations such as:

Compaction (file size tuning)
Delete file rewriting
Snapshot expiration
Partition clustering

Platforms like Dremio, AWS Glue, and Databricks provide SQL-native or automated processes (e.g., OPTIMIZE TABLE, auto-compaction) that manage these operations behind the scenes.

✅ Pros:
- Zero setup—optimization is automatic or declarative.
- Built-in cost and performance tuning.
- Reduces engineering overhead.
⚠️ Cons:
- Usually catalog-bound.
- Often restricted to Parquet format.
- Switching catalogs later requires reengineering optimization logic.

2. Manual Optimization (Bring Your Own Engine)

Open-source Iceberg supports all required lifecycle management operations—compaction, snapshot cleanup, rewrite manifests—but leaves it up to users to implement these jobs using engines like Apache Spark, Flink, Apache Amoro or Trino.

✅ Pros:
- Total freedom—no vendor lock-in.
- Can be integrated into any data pipeline or orchestration framework (Airflow, dbt, Dagster).
⚠️ Cons:
- Requires custom development and scheduling.
- Monitoring and tuning are the user's responsibility.
- Risk of misconfiguration or inconsistent maintenance across tables.

This model works well with catalogs like Apache Polaris, OneLake, or Snowflake Open Catalog, which support R/W operations but do not enforce optimization strategies.

3. Catalog-Neutral Optimization Vendors (e.g., Ryft.io)

A newer middle ground is emerging with vendors like Ryft.io, which offer catalog-agnostic optimization as a service. These platforms connect to your existing Iceberg tables—via any REST-compliant catalog—and run automated optimization jobs externally.

✅ Pros:
- Centralized, automated optimization regardless of catalog.
- Maintains interoperability and neutrality.
- Works across major cloud storage (e.g., S3, ADLS, GCS).
⚠️ Cons:
- Still a maturing category.
- Requires compatible storage (cloud object stores).
- Additional cost and integration complexity.

This is particularly valuable in multi-engine or multi-catalog environments where optimization cannot be centrally enforced but must still be automated and reliable.

Summary: The Optimization Dilemma

There is no one-size-fits-all solution:

Strategy	Best For	Primary Trade-off
Native Optimization	Simplicity, integrated platforms	Vendor lock-in, format constraints
Manual (BYO Engine)	Open source, full control	Operational complexity
Vendor-Neutral (Ryft)	Multi-cloud & multi-engine ops	Added service dependency, still emerging

Choosing an optimization strategy is not just about performance—it’s a decision about how much control you need, how much complexity you can absorb, and how much optionality you want to preserve in your architecture.

Architectural Patterns for Balancing Optimization and Interoperability

As organizations adopt Apache Iceberg REST Catalogs (IRC) to decouple compute from metadata, a recurring challenge emerges: how to balance open interoperability with the benefits of proprietary optimization. No single approach satisfies every use case. Instead, data architects are increasingly designing hybrid strategies that reflect the unique demands of their data workflows, regulatory environments, and performance SLAs.

1. Read-Only Catalogs Paired with External Optimization

Some catalogs, provide high-performance read access to Iceberg tables but restrict external writes via IRC. In these scenarios, organizations may:

Maintain a separate write-optimized catalog (e.g., Apache Polaris, Nessie, or Glue) for ingestion, transformation and optimization.
Expose tables to the read-optimized catalog after ingestion and optimization is complete.
Schedule synchronization jobs to ensure both catalogs reference consistent metadata snapshots.

This dual-catalog approach preserves the performance of engines with these restrictions while maintaining external transactional control via a neutral or R/W-capable catalog.

Pros:
- Best of both worlds: performance + flexibility.
- Avoids modifying data in restrictive environments.
Cons:
- Adds metadata orchestration complexity.
- Difficult to manage at high scale without automation.

2. Embedded Catalogs for Self-Managed Environments

Solutions like MinIO AIStor and Dremio Enterprise Catalog take a radically different approach, embedding the IRC layer directly into the object store or lakehouse platform in Dremio's case. This creates a streamlined deployment architecture for private cloud, hybrid, or air-gapped environments where full control is required.

Enables transactional Iceberg workloads without deploying a separate metadata database.
Suited for exascale, high-concurrency AI/ML pipelines.
Can be used alongside external catalogs for metadata synchronization if needed.

This model is also increasingly relevant for regulated industries or enterprises seeking on-premise lakehouse designs with built-in metadata authority.

3. Virtualized Format Interop via Metadata Translation

Microsoft OneLake, using Apache XTable, pioneers a virtualized metadata model. Instead of writing new Iceberg tables, XTable projects Iceberg-compatible metadata from Delta Lake tables in OneLake.

✅ Enables external Iceberg engines to query Delta-based data with no duplication.
🔄 Metadata is derived dynamically, enabling near real-time interop.
⚠️ Complex Iceberg-native features may be unsupported due to reliance on Delta primitives.

This architecture is ideal for organizations deeply committed to Delta Lake but wanting to provide Iceberg-compatible access for federated analytics or open-source tools.

Architectural Takeaway: Mix and Match for Your Use Case

The modern Iceberg ecosystem isn’t about picking a single vendor. Instead, it’s about selecting interoperable components that align with your architecture's performance, governance, and flexibility goals.

Scenario	Catalog Strategy	Optimization Path	Interop Balance
Cloud-native with automation	AWS Glue, Dremio	Native Automation	High (if Parquet)
Multi-engine, multi-cloud	Dremio Catalog, Snowflake Open Catalog	Built on OSS with Full Interop	Very High
Private/Hybrid cloud	MinIO AIStor or Dremio Catalog	Embedded in software for lakehouse storage or lakehouse engine	Medium–High
Stream → Lakehouse	Confluent TableFlow	Fixed strategy (snapshots)	Limited
Delta → Iceberg bridge	OneLake + XTable	Virtualized sync	High for reads

Designing an effective catalog strategy means embracing modularity—using REST interoperability as the glue while tailoring optimization and governance layers to the needs of your teams.

Conclusion: Choosing the Right Iceberg Catalog for Your Strategy

The Apache Iceberg REST Catalog ecosystem has matured into a diverse landscape of offerings—each with its own balance of interoperability, optimization capability, and vendor integration strategy. From hyperscalers to open-source initiatives, every catalog presents unique strengths and trade-offs.

At the heart of this evolution is a simple but profound architectural truth:

Compute and metadata must decouple—but performance, governance, and interoperability must still align.

🧠 Key Takeaways

If performance and simplicity are your top priorities, a native-optimization platform like Dremio, Databricks, or AWS Glue offers seamless, powerful lifecycle management.
If complete control and flexibility across tools and clouds matter more, choose a self-managed catalog like Apache Polaris and prepare to invest in your own optimization pipeline or use a neutral optimizer like Ryft.io (when on major cloud object storage) or use the OSS Apache Amoro.
If you're locked into an analytics platform like Snowflake or BigQuery, understand the implications of the differing level of Iceberg support on these platforms.

Among the platforms reviewed, Dremio strikes a rare balance: offering full Iceberg REST compatibility, native R/W support from any engine, and automated optimization, without locking users into its compute layer.

Unlike platforms that charge per API call or limit external writes, Dremio only charges for compute run through Dremio itself, meaning you can leverage external engines freely while still benefiting from the platform’s integrated catalog.

This model promotes interoperability and performance without compromise, aligning with the core principles of the Iceberg Lakehouse architecture: open metadata, multi-engine flexibility, and governed performance.

Final Thought

The Iceberg REST Catalog isn’t just an API spec, it’s the foundation for a new kind of lakehouse: open, transactional, and cloud-agnostic. Your choice of catalog defines how far you can scale without friction.

Choose wisely.

Building a Universal Lakehouse Catalog - Beyond Iceberg Tables

Alex Merced — Fri, 17 Oct 2025 09:00:00 GMT

Get Data Lakehouse Books:

Lakehouse Community:

Will be recording an episode on this topic on my podcast, so please subscribe to the podcast to not miss it (Also on iTunes and other directories)

Apache Iceberg has done something few projects manage to pull off, it created a standard. Its table format and REST-based catalog interface made it possible for different engines to read, write, and govern the same data without breaking consistency. That’s a big deal. For the first time, organizations could mix and match engines while keeping one clean, transactional view of their data.

But this success brings new expectations.

As lakehouse adoption grows, teams want more than just Iceberg tables under one roof. They want to treat all their datasets, raw Parquet files, streaming logs, external APIs, or even other formats like Delta and Hudi, with the same consistency and governance. The problem? Today’s Iceberg catalogs don’t support that. They’re built for Iceberg tables only.

So how do we move beyond that? How do we build a universal lakehouse catalog that works across engines and across formats?

Let’s explore two possible paths and what’s still missing.

Iceberg’s Success: A Case Study in Standardization

To understand where catalogs could go next, it helps to look at what made Iceberg successful in the first place.

Before Iceberg, working with data lakes was messy. You could store files in open formats like Parquet or ORC, but there was no clean way to manage schema changes, version history, or transactional consistency. Each engine had to implement its own logic, or worse, teams had to build brittle pipelines to fill in the gaps.

Iceberg changed that. It introduced:

A table format that handles schema evolution, ACID transactions, and partitioning without sacrificing openness.
A catalog interface that lets any engine discover tables and retrieve metadata in a consistent way.

These two specs, the table format and the REST catalog interface, created a plug-and-play model. Spark, Flink, Trino, Dremio, and others could all speak the same language. As a result, Iceberg became the neutral zone. No vendor lock-in, no hidden contracts.

But that neutrality came with a scope: Iceberg REST Catalog only tracks and governs Iceberg tables. If your dataset isn’t an Iceberg table, there is no modern open interoperable standard for governing and accessing. And that’s where the limitation begins.

The Problem: No Standards Beyond Iceberg

While Iceberg catalogs are tightly defined for Iceberg tables, some catalogs do allow you to register other types of datasets, raw Parquet, Delta tables, external views, or even API-based data sources.

But there’s a catch.

Each catalog handles this differently. One might use a custom registration API, another might expose a metadata file format, and yet another might treat external sources as virtual tables with limited capabilities. The result is a patchwork of behavior:

Some tools can read those datasets.
Some can't see them at all.
Others behave inconsistently depending on the engine and the catalog.

This makes interoperability fragile. What works in one engine may not work in another, even if they both support the same table format. Teams are left stitching together workarounds or writing custom integrations just to get basic access across systems.

So what’s really missing here? A standard API for non-Iceberg datasets. Something that defines:

How to register a dataset that isn't an Iceberg table.
How to describe its metadata (schema, location, stats).
How to govern access across different engines.

The big question is: where should this standard come from, and what should it look like?

Where Should the Standard Come From?

This brings us to the real crossroads: if we need a standard API for universal lakehouse catalogs, where should it come from?

There are a few possibilities:

Should it come from the Iceberg REST spec?
That would keep things in the same family and build on an existing community standard. But Iceberg’s current REST spec is tightly scoped around Iceberg tables, and expanding it to cover other data types could be a big shift and expand the project beyond what the community may be comfortable with.
Should it be defined inside a single catalog project like Polaris or Unity?
A vendor-backed project can move quickly, implement end-to-end features, and ship a working solution but then be a source of lock-in. If an open standard catalog dominates, then it becomes the home of the API standard by default.
Is it acceptable if the spec starts with a vendor?
Maybe. If that vendor drives real adoption and the API is later opened up, it can evolve into a neutral standard. But it would need wide buy-in and careful governance, to avoid becoming another moving target.

No matter how you look at it, there are really only two main paths forward:

An implementation becomes the de facto standard.
One catalog (open source or commercial) builds enough momentum that its API becomes the standard, similar to how S3 became the API for object storage.
A neutral API spec is created independently.
This would follow the Iceberg model, where the spec came first, then vendors and engines built around it.

If history teaches us anything, it’s that vendor-driven standards can create long-term friction. S3 is a good example: it's ubiquitous, but it’s also tightly bound to a single provider’s roadmap leading to a whack-a-mole like catch up game for those who support the API they have no control over. That experience shaped how the industry approached table formats, this time, the community came together around Iceberg to avoid that kind of lock-in and vendor catch-up.

So whatever path we take toward universal cataloging, the smart money is on a community standard. The only question is whether that standard comes from an existing implementation, or from a new, vendor-neutral spec that everyone agrees to follow.

Exploring the Implementation-First Path: Apache Polaris and Table Sources

If the path to a universal catalog starts with an implementation, Apache Polaris (incubating) is worth watching closely. Among the open catalog projects, Polaris stands out for two reasons:

It's built as an open implementation of the Apache Iceberg REST Catalog spec.
It's actively proposing new features to extend catalog support beyond Iceberg tables.

While Polaris already supports Iceberg tables through the standard REST interface, it's exploring how to bring non-Iceberg datasets into the same catalog. This includes both structured file-based datasets like Parquet or JSON, and unstructured data like images, PDFs, or videos.

Right now, Polaris includes a feature called Generic Tables, but a more robust proposal called Table Sources is under active discussion.

What Are Table Sources?

Discussion of this proposal on the Dev List

Table Sources are a proposed abstraction that lets Polaris register and govern external data that isn’t already an Iceberg table. Instead of forcing everything into the Iceberg format, Polaris acts as a bridge: mapping object storage locations to queryable tables using metadata services that live outside the catalog itself.

Each Table Source includes:

A name (used as the table identifier)
A source type (structured data, unstructured objects, or Iceberg metadata)
A configuration (like file format, storage location, credentials, filters, and refresh intervals)

For example:

Data Table Source: Represents structured files like Parquet or JSON. These are registered read-only tables with metadata generated by an external service.
Object Table Source: Describes unstructured data like videos or documents, exposing file metadata (size, path, modification time) in table format.
Iceberg Table Source: Adapts metadata from existing Iceberg tables stored outside Polaris.

Polaris doesn’t scan or interpret these datasets directly. Instead, Source Services, external processes, use the registered configurations to scan file systems, generate table metadata, and push it back to Polaris. This decouples the engine from the source and the catalog from the scanning logic.

At query time, engines can interact with these registered tables using the same APIs as they would for Iceberg, even though the backing data may not follow Iceberg’s spec.

Why This Matters

If adopted, the Table Source feature could give Polaris a head start as the reference implementation for a broader catalog API. It defines a reusable contract for registering external data, managing its lifecycle, and governing access, all in a way that’s decoupled from specific engines or formats.

But this also raises the bigger question: will other catalogs follow this model? Will engines adopt the same contract for recognizing external data? Or will each system continue to define its own rules?

That tension, between an evolving implementation like Polaris and the desire for an extension to the REST Catalog API standard, sets the stage for what comes next in the catalog story.

The API-First Path: Extending the Iceberg REST Catalog Spec

Now let’s explore the other side of the equation: what if instead of extending a specific implementation, we expanded the Iceberg REST Catalog specification itself?

This approach would focus on defining a neutral contract that any catalog, Polaris, Unity, Glue, or others, could implement to support more than just Iceberg tables. Rather than focusing on what a specific system can do today, it asks: what could a future REST catalog look like if it supported universal datasets by design?

One of the most interesting signs of this potential is already in the spec: the Scan Planning Endpoint.

What Is Scan Planning?

In the typical read path, an engine:

Requests a table from the catalog.
The catalog responds with the metadata location.
The engine reads the metadata files (manifests, snapshots, etc.) and plans which Parquet files to scan.

But with the Scan Planning Endpoint, the flow changes:

The engine calls the endpoint directly.
The catalog does the heavy lifting: it traverses the metadata, evaluates filters, and returns a list of data files to scan.

This makes the engine’s job simpler if the catalog and engine support the endpoint. It no longer needs to understand Iceberg’s metadata structure. It just gets files to read.

Why This Matters for Universal Catalogs

By pushing scan planning into the catalog, the spec opens the door to something bigger:

The catalog could expose non-Iceberg datasets, like Delta Lake, Hudi, or raw Parquet, and return scan plans for them.
It could also cache metadata in a relational database, avoiding repeated reads from object storage.
Engines remain agnostic to metadata formats, they just scan files.

This is a fundamental shift: the catalog becomes the query planner for metadata, not just a metadata store.

But here’s the big catch: this currently only exists on the read side.

There’s no equivalent in the spec today for the write path.

A Hypothetical Write-Side Extension

Imagine this: instead of asking the engine to write metadata files (as is required today), the engine submits a write payload to the catalog:

The namespace of the table
The table type
A list of new data files and associated summary statistics

The catalog could then:

Internally update its metadata, whether that’s JSON files, a manifest database, or some other format
Enforce governance rules
Trigger compaction or indexing tasks

In this model, the catalog fully owns metadata management for both reads and writes. Engines don’t need to understand Iceberg’s internals, or any other format’s internals. They just write and read data and delegate everything else.

The Trade-Offs

This model is clean and powerful. It simplifies engine logic and opens the door for catalogs to support any file-based dataset. But it comes at a cost:

The catalog must be deeply optimized to handle scan planning at scale.
It must support high concurrency, incremental updates, and aggressive caching.
Metadata operations become tightly coupled to catalog performance.

In other words, this model places a lot more responsibility on the catalog itself. That’s not necessarily bad, but it changes the design expectations.

Still, if the goal is to build a universal contract for working with datasets across formats, pushing more of that logic into the catalog, via a standardized API that even the major cloud vendors follow, might be the path forward.

Comparing the Two Paths: Implementation vs. API Standard

Both the Table Sources approach and the Scan Planning API model offer ways to move beyond Iceberg-only catalogs. But they take fundamentally different routes. One starts by expanding what a specific catalog can do and becomes the standard if that catalog becomes the standard. The other extends an API Spec that is already an industry standard with a narrower scope (standardizing transactions with Iceberg tables).

Let’s weigh the trade-offs.

1. Flexibility and Expressiveness

Table Sources (Implementation-first)
✅ Easier to move quickly, Polaris can prototype and evolve features as it is a younger project with a younger community that can reach consensus quicker.
✅ Can support structured and unstructured datasets with source-specific logic.
✅ Avoid the lock-in of a vendor implementation becoming the standard, since Apache Polaris is a incubating Apache Project anyone can deploy.
Scan Planning Extension (API-first)
✅ Treats all datasets as files with a metadata interface, engines don’t need to know anything about the metadata format.
✅ Opens the door for catalogs to expose Delta, Hudi, Paimon, or other sources using the same scan API.
⚠️ Metadata management becomes much more complex for the catalog, especially for large tables or real-time use cases.

In both scenarios, there is still always the question of a specific engines support for reading different file formats or metadata formats. Although, in both scenarios the catalog can still be the central listing governing access to all lakehouse datasets.

2. Governance and Control

Table Sources
✅ Catalog remains the system of record and point of governance.
✅ Supports configuration-based registration, access control, and credential vending.
⚠️ Each source type needs its own metadata strategy, increasing maintenance complexity.
Scan Planning + Write Delegation
✅ Centralizes all metadata handling, which could unify governance and simplify access rules.
⚠️ Puts more strain on catalog durability, uptime, and scalability, it's now a bigger bottleneck for reads and writes.

3. Ecosystem Alignment

Table Sources
✅ Works well for ecosystems already aligned around Polaris or compatible systems.
⚠️ Other catalogs would need to implement Polaris-compatible logic to ensure portability. (We saw catalogs adopt the Iceberg REST Spec as it become the standard, so there is precedent)
REST Spec Extension
✅ Builds on a known spec (Iceberg REST), which already has buy-in across many vendors.
✅ Keeps catalogs interchangeable if they adhere to the same read/write API contract.
⚠️ Requires coordination and consensus across the community, which can slow down adoption.

4. Developer Experience

Table Sources
✅ Clear division of responsibility: catalog governs metadata, engines execute logic.
✅ External services (source services) handle complexity and can evolve independently.
⚠️ Requires more infrastructure components to be deployed and maintained.
API Extensions
✅ Simplifies engine logic, engines just hand off files and scan what they’re told.
⚠️ Catalog APIs become more complex and require tighter validation of inputs and outputs.

Summary

In practice, both paths have strengths, and challenges. A hybrid model could even emerge: catalogs like Polaris could lead the way with working implementations, while the community formalizes an API spec based on what works.

The real question isn’t which is “better”, it’s which path brings the most durable, portable, and scalable standard to life.

Intro to Apache Iceberg with Apache Polaris and Apache Spark

Alex Merced — Thu, 16 Oct 2025 09:00:00 GMT

Get Data Lakehouse Books:

Lakehouse Community:

Modern analytics depend on flexibility. Teams want to query raw data with the same speed and reliability they expect from a warehouse. That goal led to the rise of the data lakehouse, an architecture that unifies structured and unstructured data while supporting multiple compute engines.

The lakehouse model removes silos by allowing data to live in open formats, accessible to tools like Spark, Trino, Dremio, and Flink. Interoperability becomes the foundation of this design: storage is separated from compute, and metadata lives in a shared catalog. Apache Iceberg sits at the center of this open ecosystem.

The Lakehouse and the Value of Interoperability

Traditional data systems often forced teams to choose between performance and openness. Data warehouses provided fast queries but required proprietary formats and vendor lock-in. Data lakes offered openness and low cost but lacked reliability and consistent schema management.

The lakehouse combines both. It keeps data in object storage while using open table formats like Apache Iceberg to bring reliability, version control, and transactional guarantees. This allows multiple engines to read and write the same datasets without duplication.

Interoperability is the key advantage. When organizations use open standards, they can build systems that evolve without re-platforming. Governance, lineage, and performance optimizations can be shared across tools, creating one consistent view of enterprise data.

Apache Iceberg’s Role in the Lakehouse

Apache Iceberg is the open table format that makes the lakehouse possible. It defines how large analytic tables are stored, versioned, and accessed in cloud or on-premises object storage. Iceberg tracks snapshots of data files, enabling ACID transactions, schema evolution, and time travel.

Each Iceberg table is independent of any single compute engine. Spark, Dremio, Trino, and Flink can all operate on the same tables because the format defines a consistent API for reading and writing data. This makes Iceberg a shared foundation for analytics across the open data ecosystem.

In practice, Iceberg replaces the old Hive Metastore model with a more scalable and flexible metadata structure. Tables are self-describing, and every change creates a new immutable snapshot. This design not only enables concurrency and rollback but also ensures that the same data can be reliably queried from different engines without conflict.

The Structure of an Apache Iceberg Table

An Apache Iceberg table is more than a collection of data files. It is a structured system that records every version of a dataset, allowing engines to read, write, and track changes with full transactional integrity. Understanding this structure helps explain how Iceberg enables features like time travel, schema evolution, and partition management.

At the top level, each table has a metadata directory that contains JSON files describing the current state of the table. These files point to snapshot metadata, which lists all the data files that make up the current version. Each snapshot references one or more manifest lists, and each manifest list points to multiple manifest files. Manifest files contain the actual list of data files, typically Parquet, ORC, or Avro, along with partition information and statistics.

Every time you insert, delete, or update data, Iceberg creates a new snapshot without rewriting the existing files. This immutable design ensures that multiple users and engines can safely interact with the same table at the same time. It also makes rollback and version tracking possible, since previous snapshots are always preserved until explicitly expired.

Iceberg also introduces a flexible approach to partitioning. Instead of static directories like in Hive, Iceberg uses partition transforms that record logical rules, such as bucket(8, id) or months(order_date), directly in metadata. This allows the table to manage partitions dynamically, improving query performance while keeping partitioning transparent to users.

Together, these components form a self-contained and versioned system that makes object storage behave like a transactional database. In the next section, you’ll set up an environment using Apache Polaris and Apache Spark to see how this structure works in practice.

Setting Up the Environment

To explore how Apache Iceberg works in practice, you’ll use a local setup that includes three components: Apache Polaris, MinIO, and Apache Spark. Polaris will serve as the catalog that manages Iceberg metadata, MinIO will act as your S3-compatible storage system, and Spark will be your compute engine for creating and querying tables.

A catalog in Iceberg defines where tables are stored and how their metadata is managed. It is responsible for keeping track of namespaces, table locations, and access control. Apache Polaris provides an open-source implementation of an Iceberg catalog that exposes a REST API for managing these operations. Polaris also adds governance features, authentication, roles, and permissions, making it more than just a metadata store.

Within Polaris, users and services are represented as principals, each with unique credentials that determine what they can access. You can assign roles and privileges to principals, giving them permission to create, update, or query catalogs and tables. This design allows multiple tools to share a single governed catalog while maintaining secure, fine-grained access.

Starting the Environment

Clone the quickstart repository and start the environment using Docker Compose:

git clone https://github.com/AlexMercedCoder/Apache-Polaris-Apache-Iceberg-Minio-Spark-Quickstart.git
cd Apache-Polaris-Apache-Iceberg-Minio-Spark-Quickstart
docker compose up -d

This will launch:

Polaris on port 8181 (catalog API)
MinIO on ports 9000 and 9001 (S3 and web console)
Spark with Jupyter Notebook on port 8888

You can verify that all containers are running with:

docker ps

Once they’re up, open the Jupyter Notebook interface by visiting http://localhost:8888. Create a new Python notebook and copy the contents of bootstrap.py from the repository into a cell. Running this script will bootstrap Polaris by:

Creating two catalogs—lakehouse and warehouse—that point to MinIO buckets.
Defining a principal with access credentials.
Assigning roles and granting full permissions to that principal.

When the script completes, it prints a ready-to-use Spark configuration block with all the connection details. You’ll use that configuration in the next section to create and manage Iceberg tables through Polaris.

Creating Iceberg Tables

With Polaris bootstrapped and Spark connected, you’re ready to start working with Iceberg tables. The tables you create will live in the polaris.db namespace, with their data stored in your MinIO buckets. All catalog and permission management will happen automatically through Polaris.

Before you begin creating tables, make sure Spark is configured to connect to Polaris. When you ran bootstrap.py, the script printed out a Spark configuration block similar to the example below. This block contains the packages, catalog URI, warehouse name, and your principal’s credentials. Copy this block into a cell in your Jupyter Notebook and run it to initialize your Spark session.

# Spark configuration for catalog: lakehouse
from pyspark.sql import SparkSession

spark = (SparkSession.builder
    .config("spark.jars.packages", "org.apache.polaris:polaris-spark-3.5_2.13:1.1.0-incubating,org.apache.iceberg:iceberg-aws-bundle:1.10.0,io.delta:delta-spark_2.12:3.3.1,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.0")
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.polaris", "org.apache.polaris.spark.SparkCatalog")
    .config("spark.sql.catalog.polaris.uri", "http://polaris:8181/api/catalog")
    .config("spark.sql.catalog.polaris.warehouse", "lakehouse")
    .config("spark.sql.catalog.polaris.credential", "{client_id}:{client_secret}")
    .config("spark.sql.catalog.polaris.scope", "PRINCIPAL_ROLE:ALL")
    .config("spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation", "vended-credentials")
    .config("spark.sql.catalog.polaris.token-refresh-enabled", "true")
    .getOrCreate())

spark.sql("CREATE NAMESPACE IF NOT EXISTS polaris.db").show()
spark.sql("CREATE TABLE IF NOT EXISTS polaris.db.example (name STRING)").show()
spark.sql("INSERT INTO polaris.db.example VALUES ('example value')").show()
spark.sql("SELECT * FROM polaris.db.example").show()

The client_id and client_secret values should be filled in with the code printed at the end of running your bootstrap script. Once the Spark session starts, you’ll be able to issue SQL commands directly against Polaris.

Creating a Basic Table

Start by setting your working namespace and creating a simple unpartitioned table:

# eliminates the need to prefix table names with the namespace polaris.db
spark.sql("USE polaris.db")

spark.sql("""
CREATE TABLE customers (
    id INT,
    name STRING,
    city STRING
)
USING iceberg
""")

This creates a new Iceberg table tracked by Polaris. You can confirm its existence by listing all tables in the namespace:

spark.sql("SHOW TABLES IN polaris.db").show()

Now open the MinIO console at http://localhost:9001 (admin/password are the credentials) and explore the lakehouse bucket—you’ll see a new folder structure created for your table. This directory contains the Parquet data files and the metadata that Polaris manages.

Partitioned Tables

Partitioning helps improve performance by organizing data into logical groups. Iceberg’s partition transforms let you define flexible strategies without depending on directory names.

Partition by a single column:

spark.sql("""
CREATE TABLE polaris.db.sales (
    sale_id INT,
    product STRING,
    quantity INT,
    city STRING
)
USING iceberg
PARTITIONED BY (city)
""")

Partition by time:

spark.sql("""
CREATE TABLE polaris.db.orders (
    order_id INT,
    customer_id INT,
    order_date DATE,
    total DECIMAL(10,2)
)
USING iceberg
PARTITIONED BY (months(order_date))
""")

Partition by hash buckets for even data distribution:

spark.sql("""
CREATE TABLE polaris.db.transactions (
    txn_id BIGINT,
    user_id BIGINT,
    amount DOUBLE
)
USING iceberg
PARTITIONED BY (bucket(8, user_id))
""")

Each strategy changes how Iceberg organizes data, but all are tracked as metadata—not directories—making future changes safe and reversible.

After creating your tables, return to the MinIO console to explore the results. You’ll notice new directories and metadata files representing the structure of each table. These files are created and tracked automatically by Polaris, ensuring that every write, update, and schema change remains consistent across all engines that connect to the catalog.

Inserting Data

Once your tables are created, you can begin inserting and modifying data through Spark. Every write operation, whether it’s an insert, update, or delete, creates a new snapshot in Iceberg. Each snapshot represents a consistent view of your table at a specific point in time and is recorded in Polaris’s metadata catalog.

Start with a simple insert:

spark.sql("""
INSERT INTO polaris.db.customers VALUES
(1, 'Alice', 'New York'),
(2, 'Bob', 'Chicago'),
(3, 'Carla', 'Boston')
""")

After this insert, open the MinIO console and look inside the lakehouse bucket under polaris/db/customers. You’ll see a new folder structure containing Parquet files and Iceberg metadata files (metadata.json, snapshots, and manifests). Each write creates new files rather than overwriting existing ones, which is how Iceberg maintains atomic transactions and rollback capabilities.

Inserting into Partitioned Tables

If you created partitioned tables earlier, Iceberg will automatically place data into the correct partitions based on your table definition:

spark.sql("""
INSERT INTO polaris.db.sales VALUES
(101, 'Laptop', 5, 'New York'),
(102, 'Tablet', 3, 'Boston'),
(103, 'Phone', 7, 'Chicago')
""")

To confirm partitioning, you can check MinIO. Each partition value (in this case, city) will have its own subdirectory. Iceberg manages these directories automatically through metadata, keeping partitioning invisible to end users.

Working with Larger Datasets

For larger datasets, you can also write directly from a DataFrame:

data = [(201, 'Monitor', 2, 'Denver'),
        (202, 'Keyboard', 10, 'Austin')]

df = spark.createDataFrame(data, ['sale_id', 'product', 'quantity', 'city'])
df.writeTo("polaris.db.sales").append()

This method is efficient for batch operations and ensures your Spark DataFrames integrate cleanly with Iceberg’s transaction system.

Each time you perform a write, Polaris updates the catalog with a new snapshot ID. These snapshots allow you to query your table as it existed at any point in time, a capability you’ll explore later in the section on time travel.

For now, review the lakehouse bucket in MinIO after each insert to see how Iceberg adds new Parquet and metadata files. Each transaction tells a story of how the table evolves over time, tracked and governed by Polaris.

Update, Delete, and Merge Into

Apache Iceberg provides full ACID transaction support, allowing you to update, delete, and merge data safely. Each of these operations creates a new snapshot while preserving older versions of the table, giving you consistent rollback and auditing capabilities. Polaris tracks these changes in its catalog so that every engine accessing the table sees a consistent state.

Updating Data

Use UPDATE to modify existing records. For example, if one of your customers relocates:

spark.sql("""
UPDATE polaris.db.customers
SET city = 'San Francisco'
WHERE name = 'Alice'
""")

This statement creates a new snapshot that replaces the affected rows with updated data. Iceberg performs this by rewriting only the data files that contain the changed rows, which keeps transactions efficient even at scale.

Deleting Data

You can delete records using a standard DELETE statement:

spark.sql("""
DELETE FROM polaris.db.customers
WHERE name = 'Bob'
""")

After running this command, open the MinIO console and look at the customers directory in the lakehouse bucket. You’ll notice new Parquet and metadata files have appeared, Iceberg never mutates existing files. Instead, it writes new ones and updates the catalog’s snapshot metadata through Polaris.

Merging Data (Upserts)

The MERGE INTO command allows you to perform upserts, merging new records with existing data based on a matching key. This is especially useful when syncing incremental updates from another source.

First, create a temporary table or view that holds your new data:

spark.sql("""
CREATE OR REPLACE TEMP VIEW updates AS
SELECT 1 AS id, 'Alice' AS name, 'Seattle' AS city
UNION ALL
SELECT 4 AS id, 'Dana' AS name, 'Austin' AS city
""")

Then merge it into your main table:

spark.sql("""
MERGE INTO polaris.db.customers AS target
USING updates AS source
ON target.id = source.id
WHEN MATCHED THEN UPDATE SET city = source.city
WHEN NOT MATCHED THEN INSERT *
""")

After the merge completes, Polaris will record a new snapshot in the catalog. You can query the customers.history or customers.snapshots metadata tables to see when and how the change occurred.

Each of these operations, UPDATE, DELETE, and MERGE INTO, produces new files in MinIO and new snapshots in Polaris. This versioned structure ensures your tables remain fully auditable. Take a moment to check the lakehouse bucket again after running each command. You’ll see Iceberg’s design in action: immutable data files, evolving metadata, and transparent version control, all orchestrated through Polaris.

Altering Partition Scheme

Over time, your table’s partitioning strategy may need to change as data grows or query patterns evolve. Apache Iceberg allows you to alter partition schemes safely, without rewriting existing files. This flexibility is one of Iceberg’s biggest advantages over traditional data lake formats. All changes are tracked by Polaris, ensuring that the catalog always reflects the current partition structure.

Suppose your sales table is currently partitioned by city. If queries start filtering by product instead, you can modify the table’s partitioning to better suit that use case. Start by dropping the old partition field:

spark.sql("""
ALTER TABLE polaris.db.sales
DROP PARTITION FIELD city
""")

Then add a new partition field:

spark.sql("""
ALTER TABLE polaris.db.sales
ADD PARTITION FIELD bucket(8, product)
""")

This change affects only future writes. Existing data remains organized by the previous partition scheme, while new records follow the new one. Iceberg’s metadata model keeps track of both versions, so queries continue to return complete results without manual migration.

To verify your table’s current partitioning, run:

spark.sql("SHOW PARTITIONS polaris.db.sales").show()

You can also view the table’s partition history through the metadata tables:

spark.sql("SELECT * FROM polaris.db.sales.partitions").show()

After altering partition fields, try inserting new records and observe how Iceberg places them into new directories in MinIO. Open the lakehouse bucket in the MinIO console, navigate to your sales folder, and you’ll see both the old and new partition structures coexisting under the same table. Polaris ensures the catalog references all of them correctly.

This feature makes partition evolution seamless. You can adapt to new data patterns or performance needs without downtime, data duplication, or complex ETL steps. In the next section, you’ll learn how to explore Iceberg’s built-in metadata tables and use time travel to query historical versions of your data.

Metadata Tables and Time Travel

Apache Iceberg doesn’t just store data—it stores the entire history of your data. Every write operation creates a new snapshot, and every snapshot is tracked in the table’s metadata. These metadata tables give you full visibility into how your data changes over time. Because Polaris manages the catalog, you can query these tables from any engine that connects to it, ensuring a unified and governed view of your data lifecycle.

Exploring Metadata Tables

Each Iceberg table automatically includes several metadata tables that you can query just like normal tables:

history – shows when snapshots were created.
snapshots – lists snapshot IDs and timestamps.
files – lists all data and manifest files in each snapshot.
manifests – details how files are grouped and filtered.

You can explore them with Spark SQL:

spark.sql("SELECT * FROM polaris.db.sales.history").show()
spark.sql("SELECT * FROM polaris.db.sales.snapshots").show()
spark.sql("SELECT * FROM polaris.db.sales.files").show()

These tables reveal every version of your dataset, what files were written, when they were created, and by which operation. You can use this information for auditing, debugging, or optimizing table performance.

Querying Past Versions with Time Travel

Because Iceberg stores all historical snapshots, you can query data as it existed at a specific point in time. You can travel through time using either a snapshot ID or a timestamp.

First, identify a snapshot ID from the snapshots table:

spark.sql("SELECT snapshot_id, committed_at FROM polaris.db.sales.snapshots").show()

Then query that version:

spark.read.option("snapshot-id", "<snapshot_id>").table("polaris.db.sales").show()

Alternatively, you can query the table as it existed at a given timestamp:

spark.read.option("as-of-timestamp", "2025-10-10T12:00:00").table("polaris.db.sales").show()

This ability to reproduce historical states makes Iceberg ideal for debugging ETL processes, reproducing analytics, or auditing compliance-related datasets.

Seeing It in MinIO

Each time you insert, update, or delete data, Iceberg records a new snapshot. Open the lakehouse bucket in MinIO and navigate through your table directories—you’ll notice subdirectories under metadata/ representing each snapshot and manifest. Every change to your data produces new metadata and data files, which together describe the complete history of your table.

Iceberg’s metadata and time travel capabilities, combined with Polaris’s catalog management, give you full traceability and reproducibility. In the next section, you’ll learn how to keep your tables healthy by compacting small files and expiring old snapshots.

Compaction and Snapshot Expiration

As you run inserts, updates, and merges, Iceberg continuously creates new data and metadata files. Over time, this can lead to many small files and obsolete snapshots. To maintain performance and control storage costs, Iceberg provides built-in maintenance operations for compaction and snapshot expiration. With Polaris managing the catalog, these optimizations remain consistent and trackable across all compute engines that access your tables.

Compacting Small Files

Small files are common in streaming or frequent batch ingestion workflows. Iceberg can merge them into fewer, larger files using the rewrite_data_files procedure. This reduces overhead during query planning and execution.

Run the following command from Spark to compact your table:

spark.sql("""
CALL polaris.system.rewrite_data_files('polaris.db.sales')
""")

You can also target specific partitions or filter files by size:

spark.sql("""
CALL polaris.system.rewrite_data_files(
  table => 'polaris.db.sales',
  options => map('min-input-files', '4', 'max-concurrent-rewrites', '2')
)
""")

After compaction, check your lakehouse bucket in MinIO. You’ll notice fewer Parquet files, each larger in size. Iceberg automatically updates manifests and metadata files so that queries continue to return accurate results with better performance.

Expiring Old Snapshots

Every Iceberg operation creates a snapshot. Over time, unused snapshots can accumulate, consuming metadata space and storage. Iceberg allows you to remove these safely using the expire_snapshots procedure.

For example, to remove snapshots older than seven days:

spark.sql("""
CALL polaris.system.expire_snapshots(
  table => 'polaris.db.sales',
  older_than => TIMESTAMPADD(DAY, -7, CURRENT_TIMESTAMP)
)
""")

You can also specify how many snapshots to retain regardless of age:

spark.sql("""
CALL polaris.system.expire_snapshots(
  table => 'polaris.db.sales',
  retain_last => 5
)
""")

Polaris automatically tracks the catalog state after expiration, ensuring that all compute engines accessing the table remain synchronized with the current set of snapshots.

Monitoring with Metadata Tables

After compaction or expiration, you can verify changes using the metadata tables:

spark.sql("SELECT * FROM polaris.db.sales.snapshots").show()
spark.sql("SELECT * FROM polaris.db.sales.manifests").show()

You’ll see fewer manifests and snapshots, confirming that Iceberg has reclaimed space and simplified query planning.

Maintenance operations like compaction and snapshot expiration help keep your Iceberg tables fast and cost-efficient. Combined with Polaris’s centralized catalog, these operations stay consistent across all connected engines. Whether you’re using Spark, Dremio, Trino, or Flink, Polaris ensures a single source of truth for your Iceberg metadata, making performance optimization and governance effortless.

Writing Efficiently to Apache Iceberg with Spark

When working with Apache Iceberg tables in Spark, how you write data has a major impact on performance, metadata growth, and maintenance frequency. Iceberg is designed for incremental writes and schema evolution, but inefficient write patterns—like frequent small updates or poor partitioning—can lead to excessive snapshots and small files. By tuning Spark and table-level settings, you can reduce the need for costly compaction and keep your tables query-ready.

Optimize File Size and Shuffle Configuration

Each write produces data files that Spark generates in parallel tasks. If your partitions are too small or the number of shuffle tasks is too high, Spark creates many tiny files, increasing metadata overhead and slowing queries. To control this, adjust Spark’s shuffle and output configurations before writing:

spark.conf.set("spark.sql.shuffle.partitions", 8)
spark.conf.set("spark.sql.files.maxRecordsPerFile", 5_000_000)

These settings reduce the number of output files per job and encourage larger Parquet files (typically 128–512 MB each). You can also call .coalesce() or .repartition() before writes to further control file output:

df.coalesce(8).writeTo("polaris.db.sales").append()

Balanced partitioning and file sizing keep your table fast and avoid unnecessary metadata bloat.

Use Table Properties to Guide Iceberg Behavior

Iceberg provides table-level configuration options that influence how data is written, compacted, and validated. You can define them during table creation or later using ALTER TABLE.

For example:

CREATE TABLE polaris.db.sales (
  id BIGINT,
  region STRING,
  sale_date DATE,
  amount DOUBLE
)
USING iceberg
PARTITIONED BY (days(sale_date))
TBLPROPERTIES (
  'write.target-file-size-bytes'='268435456',  -- 256 MB target file size
  'commit.manifest-merge.enabled'='true',       -- reduces manifest churn
  'write.distribution-mode'='hash',             -- distributes data evenly
  'write.merge.mode'='copy-on-write'            -- ensures clean updates
);

You can also modify these settings later:

ALTER TABLE polaris.db.sales SET TBLPROPERTIES (
  'write.target-file-size-bytes'='536870912'  -- 512 MB
);

Setting appropriate table properties ensures consistent behavior across all engines—Spark, Dremio, or Flink—that share your Polaris catalog.

Batch and Append Data Strategically

Each write in Iceberg creates a new snapshot. If your application writes too frequently (e.g., per record or small microbatch), metadata grows quickly and queries slow down. Instead, buffer data into larger batches before committing:

batch_df.writeTo("polaris.db.sales").append()

If you need streaming ingestion, tune the microbatch trigger interval and commit size. A five-minute trigger often balances latency and table stability better than writing every few seconds.

For update-heavy workloads, consider using Merge-Into operations periodically rather than constant row-level updates:

MERGE INTO polaris.db.sales t
USING updates u
ON t.id = u.id
WHEN MATCHED THEN UPDATE SET amount = u.amount
WHEN NOT MATCHED THEN INSERT *

This avoids snapshot sprawl and makes compaction less frequent.

Align Partitioning with Query Patterns

Good partitioning reduces the number of files scanned per query. Avoid partitioning by high-cardinality columns like user_id. Instead, use transforms that group data efficiently:

ALTER TABLE polaris.db.sales REPLACE PARTITION FIELD sale_date WITH days(sale_date)

Or combine multiple transforms for balance:

CREATE TABLE polaris.db.sales (
  id BIGINT,
  region STRING,
  sale_date DATE,
  amount DOUBLE
)
USING iceberg
PARTITIONED BY (bucket(8, region), days(sale_date))

These partitioning rules make pruning effective and improve both reads and writes.

Tune Commit and Validation Settings

For large write jobs, commit coordination and validation can also affect performance. Iceberg supports asynchronous manifest merging and snapshot cleanup to reduce contention:

ALTER TABLE polaris.db.sales SET TBLPROPERTIES (
  'commit.manifest-merge.enabled'='true',
  'commit.retry.num-retries'='5',
  'write.distribution-mode'='hash'
);

These settings help large concurrent writers (for example, in Spark and Flink) commit safely to the same table without conflicts.

Efficient Iceberg write patterns come from tuning Spark and table properties together. Use larger file targets, consistent partitioning, and controlled batch sizes to minimize small files and snapshot churn. By applying these strategies, your Iceberg tables will stay lean and performant—reducing the need for manual compaction or cleanup. Combined with Apache Polaris, your catalog enforces consistent governance, authentication, and metadata management across every compute engine in your lakehouse.

12. Understanding How Polaris Manages Your Iceberg Tables

Once you have optimized your write strategy, it’s worth understanding what happens behind the scenes when you write data into Iceberg tables through Apache Polaris. Polaris acts as a centralized catalog—responsible for managing all metadata about your tables, snapshots, and permissions—ensuring that every write or read operation is consistent across tools like Spark, Dremio, Trino, and Flink.

When Spark writes to an Iceberg table using Polaris, the process goes beyond simply saving files to MinIO or S3. Each commit updates a snapshot—a precise record of table state including data files, manifests, and partition metadata. Polaris stores the metadata pointers, enforces ACID guarantees, and validates that every write operation maintains table consistency.

Coordinating Metadata and Commits

Each write to an Iceberg table involves several steps:

Spark writes data files (usually in Parquet format) to the storage layer, such as MinIO.
Spark generates a manifest list describing these new data files.
The Iceberg REST client, through Polaris, updates the catalog’s metadata location and commits the new snapshot.
Polaris enforces isolation and conflict detection to ensure concurrent writers don’t overwrite each other’s work.

Because Polaris manages these metadata transactions centrally, it becomes the single source of truth for all engines. This makes cross-engine interoperability reliable—Spark can write data, and Dremio or Trino can query it immediately without any manual refresh.

Governance and Security

Polaris also introduces a security layer around Iceberg. Instead of embedding access keys or S3 credentials in your Spark jobs, Polaris can vend temporary credentials that enforce fine-grained access control. Each principal and catalog role determines what operations are allowed, ensuring that users and jobs interact only with the tables they are permitted to modify or query.

This approach decouples data governance from compute infrastructure. You can manage permissions, audit access, and rotate credentials—all directly through Polaris—while still using open data lakehouse standards like Apache Iceberg.

Automatic Table Optimization in Dremio

If you use Dremio’s integrated catalog (built on Polaris), you also gain automated table optimization. Dremio monitors data size, file counts, and snapshot churn, then automatically runs compaction and metadata cleanup as needed. It maintains your Iceberg tables in an optimized state without requiring manual Spark procedures.

That means you can focus on analytics, while Dremio and Polaris handle governance, credential management, and metadata consistency across all your compute platforms.

With this understanding, you now have a complete end-to-end view of how Apache Spark and Apache Polaris work together to maintain a modern, open lakehouse. From efficient write strategies to managed metadata and automated optimization, you can confidently scale your Iceberg data platform knowing it’s governed, interoperable, and future-proof.

Next Steps and Expanding Your Lakehouse

Now that you’ve successfully set up Apache Polaris with Spark and Iceberg on your local machine, you’ve built a foundation for exploring the broader lakehouse ecosystem. This environment not only lets you understand Iceberg’s core table mechanics but also shows how a catalog like Polaris centralizes governance, metadata, and access control—key components of an interoperable lakehouse architecture.

Connect More Compute Engines

Polaris is designed to work seamlessly across multiple compute engines. Once your Iceberg tables are registered in Polaris, you can connect tools such as:

Dremio – Query and optimize Iceberg tables visually through its integrated Polaris-based catalog.
Trino – Use Polaris as a REST-based catalog for federated queries across your data lake.
Flink – Stream data into Iceberg tables managed by Polaris for real-time analytics.
DuckDB or Python (PyIceberg) – Interact directly with Iceberg tables for lightweight local exploration.

Each of these engines communicates through the same Polaris REST interface, ensuring that all metadata and access control remain consistent, no matter where you query from.

Experiment with Advanced Iceberg Features

Once you’re comfortable with the basics, try exploring Iceberg’s advanced capabilities:

Schema Evolution – Add, rename, or delete columns without rewriting data.
Row-Level Deletes – Use deletion vectors for efficient, fine-grained record removal.
Table Branching and Tagging – Experiment safely with data changes using versioned metadata.
Snapshot Isolation – Test concurrent writes to understand Iceberg’s transaction model.

These features are fully tracked by Polaris, giving you a reliable, auditable history of every change.

Extend with Automation and Orchestration

You can also automate your setup and maintenance workflows:

Use Airflow or Cron to run the bootstrap.py script on a schedule, ensuring consistent initialization of catalogs and principals.
Create periodic compaction or snapshot expiration jobs using Spark SQL.
Deploy your Polaris setup in Kubernetes using Helm or Docker Compose for multi-user testing environments.

Prepare for Cloud or Hybrid Deployment

The setup you’ve built locally with MinIO can easily extend to real cloud storage systems. Replace your MinIO endpoint with S3, GCS, or Azure Blob credentials, and Polaris will manage your Iceberg tables just as before—using the same metadata model and APIs.

This local-to-cloud continuity is one of the greatest advantages of Iceberg and Polaris: your data architecture can scale from a personal laptop demo to a full production lakehouse without refactoring or vendor lock-in.

Wrapping Up

You’ve now seen how Apache Iceberg, Apache Polaris, and Apache Spark work together to form a robust, open lakehouse. Through this hands-on setup, you’ve learned how to:

Write and optimize Iceberg tables in Spark.
Manage metadata, catalogs, and access through Polaris.
Explore advanced Iceberg features safely and efficiently.

For larger-scale deployments—or if you want automated optimization, integrated governance, and performance acceleration—explore Dremio’s Intelligent Lakehouse Platform, which builds directly on Apache Polaris and Iceberg to deliver a unified, self-service analytics experience.

The State of Apache Iceberg v4 - October 2025 Edition

Alex Merced — Tue, 14 Oct 2025 09:00:00 GMT

Get Data Lakehouse Books:

Lakehouse Community:

Apache Iceberg has come a long way since its early days of bringing reliable ACID transactions and schema evolution to the data lake. It helped teams move beyond brittle Hive tables and built the foundation for modern lakehouse architectures. But with wider adoption came new challenges—especially as workloads shifted from batch-heavy pipelines to streaming ingestion, faster commits, and more interactive use cases.

That pressure has exposed some cracks in the foundation. Write-heavy applications hit metadata bottlenecks. Query planners struggle with inefficient stats. Teams managing large tables face complex migrations due to rigid path references.

The Apache Iceberg community has responded with a set of focused, forward-looking proposals that make up the v4 specification. These aren’t just incremental tweaks. They represent a clear architectural shift toward scalability, operational simplicity, and real-time readiness.

In this post, we’ll walk through the key features proposed for Iceberg v4, why they matter, and what they mean for data engineers, architects, and teams building at scale.

The New Iceberg Vision: Performance Meets Portability

Apache Iceberg was initially built for reliable batch analytics on cloud object storage. It solved core problems like schema evolution, snapshot isolation, and data consistency across distributed files. That foundation made it a favorite for building open data lakehouses.

But today’s data platforms are evolving fast. Teams are mixing streaming and batch. Ingest rates are higher. Table sizes are bigger. Query expectations are more demanding. Managing metadata at scale has become one of the biggest friction points.

The proposals in Iceberg v4 address these shifts head-on. Together, they aim to:

Reduce write overhead so commits scale with ingestion speed
Improve query planning by making metadata easier to scan and use
Simplify operations like moving, cloning, or backing up tables

In short, Iceberg is being re-tuned for modern workloads—ones that demand both speed and flexibility. The v4 changes aren’t just about performance. They’re about making Iceberg easier to run, easier to optimize, and better suited for the next generation of data systems.

Proposal 1: Single-File Commits – Cutting Down Metadata Overhead

Every commit to an Iceberg table today creates at least two new metadata files: one for the updated manifest list, and another for any changed manifests. In fast-moving environments—like streaming ingestion or micro-batch pipelines—this adds up quickly.

The result? Write amplification. For every small data change, there’s a burst of I/O to update metadata. Over time, this leads to thousands of small metadata files, bloated storage, and a slowdown in commit throughput. Teams often have to schedule compaction jobs to clean up the data.

The v4 proposal introduces Single-File Commits, a new way to consolidate all metadata changes into a single file per commit. This reduces:

The number of file system operations per commit
The coordination overhead for concurrent writers
The need for frequent compaction

By minimizing I/O and simplifying commit logic, this change unlocks faster ingestion and makes Iceberg friendlier to real-time workflows. It also means fewer moving parts to manage and fewer edge cases to debug in production.

Proposal 2: Parquet for Metadata – Smarter Query Planning

Today, Iceberg stores metadata files—like manifests and manifest lists—in Apache Avro, a row-based format. While this made sense early on, it’s become a bottleneck for query performance.

Why? Because most query engines don’t need every field in the metadata. For example, if a planner wants to filter files based on a column’s min and max values, it only needs that one field. But with Avro, it has to read and deserialize entire rows just to access a few columns.

The proposed change in Iceberg v4 is to use Parquet instead of Avro for metadata files. Since Parquet is a columnar format, engines can:

Read only the fields they need
Skip over irrelevant parts of the file
Load metadata faster and use less memory

This shift isn’t just about speed—it enables smarter planning. Engines can project just the stats they care about, sort and filter more effectively, and better optimize execution plans. It’s a small architectural change with a big ripple effect across the query lifecycle.

Proposal 3: Column Statistics Overhaul – Better Skipping, Smarter Queries

Metadata isn't just about file paths—it's also about understanding what’s inside each file. Iceberg uses column-level statistics to help query engines skip files that don’t match filter conditions. But the current stats format has limitations that hold back performance.

Right now, statistics are:

Flat and untyped, with no indication of data type
Stored as generic key-value pairs
Lacking detail on things like null counts or nested fields

These gaps make it hard for query planners to fully optimize their logic. For example, it's difficult to distinguish between a missing value and a null, or to reason about nested data structures like structs and arrays.

The v4 spec proposes a redesigned statistics format with:

Type information for every stat
Projectable structures for selective reads
Support for more detailed metrics, including null counts and nested fields

This richer structure enables more precise file pruning and better cost-based optimization. Engines can make smarter decisions about which files to read and which filters to push down—leading to faster queries, less I/O, and improved overall performance.

Proposal 4: Relative Paths – Making Tables Portable Again

In current versions of Iceberg, metadata files store absolute file paths. That might seem fine at first—until you try to move a table.

If you change storage accounts, rename a bucket, or migrate between environments, every path in every metadata file becomes invalid. Fixing that means scanning and rewriting all metadata—an expensive, error-prone operation that often requires a distributed job.

The v4 proposal introduces support for relative paths in metadata. Instead of locking a table to a fixed storage location, file references are stored relative to a base URI defined in the table metadata.

This change unlocks several real-world benefits:

Simpler migrations across cloud regions or storage platforms
Easier disaster recovery with portable backups
Less brittle operations when storage configurations evolve

Relative paths decouple the logical structure of a table from its physical location. That means fewer rewrites, less maintenance overhead, and more flexibility when managing Iceberg tables at scale.

Iceberg’s Direction: Toward Operational Simplicity

Taken together, these proposals reflect a clear shift in how the Iceberg community is thinking about the format—not just as a technical layer, but as an operational foundation for modern data platforms.

Here’s what’s changing:

From batch-first to real-time ready: Single-file commits and smarter stats make Iceberg more suitable for streaming ingestion and low-latency use cases.
From fixed to flexible: Relative paths reduce the coupling between metadata and storage, making operations like migration and backup less painful.
From rigid to optimized: Moving to columnar metadata and richer statistics gives query engines more room to optimize without heavy lifting.

This is Iceberg growing up.

The format has always prioritized correctness and openness. Now it’s doubling down on speed, scalability, and ease of use—especially for teams managing hundreds or thousands of tables across dynamic environments.

Whether you're building AI pipelines, federated queries, or traditional dashboards, these changes aim to reduce the friction and complexity of working with large-scale tables. It’s about making Iceberg not just powerful, but practical.

A Glimpse Ahead: v4 in Context

Just a few months ago, Apache Iceberg v3 was approved—bringing meaningful improvements to the table format. That release introduced new data types, deletion vectors, and other enhancements that expanded what Iceberg can represent and how it supports evolving workloads.

Right now, the ecosystem is heads-down implementing v3 features across engines, catalogs, and query layers. You’ll see more engines support features like row-level deletes and richer data modeling as v3 adoption matures.

The proposals for v4 aren’t intended to replace that momentum—they build on it.

Think of v3 as expanding what Iceberg can do. V4 focuses on how efficiently and cleanly it can perform the task. These early discussions around v4 offer a forward-looking roadmap for how Iceberg will continue to evolve—toward higher throughput, better portability, and more brilliant query performance.

While these changes are still in the design and discussion phase, they signal where Iceberg is heading. For data teams investing in the lakehouse stack today, it’s reassuring that the foundation will only get stronger over time.

The Ultimate Guide to Open Table Formats - Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

Alex Merced — Wed, 24 Sep 2025 09:00:00 GMT

Get Data Lakehouse Books:

Join the Data Lakehouse Community Data Lakehouse Blog Roll

Modern lakehouse stacks live or die by how they manage tables on cheap, scalable object storage. That “how” is the job of open table formats, the layer that turns piles of Parquet/ORC files into reliable, ACID-compliant tables with schema evolution, time travel, and efficient query planning. If you’ve ever wrestled with brittle Hive tables, small-file explosions, or “append-only” lakes that can’t handle updates and deletes, you already know why this layer matters.

In this guide, we’ll demystify the five formats you’re most likely to encounter:

Apache Iceberg - snapshot- and manifest–driven, engine-agnostic, fast for large-scale analytics.
Delta Lake - transaction-log–based, deeply integrated with Spark/Databricks, strong batch/stream unification.
Apache Hudi - built for upserts, deletes, and incremental processing; flexible COW/MOR modes.
Apache Paimon - streaming-first with an LSM-like design for high-velocity updates and near-real-time reads.
DuckLake - a fresh, catalog-centric approach that uses a relational database for metadata (SQL all the way down).

We’ll start beginner-friendly, clarifying what a table format is and why it’s essential, then progressively dive into expert-level topics: metadata internals (snapshots, logs, manifests, LSM levels), row-level change strategies (COW, MOR, delete vectors), performance trade-offs, ecosystem support (Spark, Flink, Trino/Presto, DuckDB, warehouses), and adoption trends you should factor into your roadmap.

By the end, you’ll have a practical mental model to choose the right format for your workloads, whether you’re optimizing petabyte-scale analytics, enabling near-real-time CDC, or simplifying your metadata layer for developer velocity.

Why Open Table Formats Exist

Before diving into each format, it’s worth understanding why open table formats became necessary in the first place.

Traditional data lakes, built on raw files like CSV, JSON, or Parquet, were cheap and scalable, but brittle. They had no concept of transactions, which meant if two jobs wrote data at the same time, you could easily end up with partial or corrupted results. Schema evolution was painful, renaming or reordering columns could break queries, and updating or deleting even a single row often meant rewriting entire partitions.

Meanwhile, enterprises still needed database-like features, updates, deletes, versioning, auditing, on their data lakes. That tension set the stage for open table formats. These formats layer metadata and transaction protocols on top of files to give the data lake the brains of a database while keeping its open, flexible nature.

In practice, open table formats deliver several critical capabilities:

ACID Transactions: Ensure reliability for concurrent reads and writes.
Schema Evolution: Add, drop, or rename fields without breaking downstream consumers.
Time Travel: Query data as it existed at a specific point in time for auditing or recovery.
Efficient Queries: Push down filters and prune partitions/files using metadata rather than scanning everything.
Row-Level Mutations: Support upserts, merges, and deletes on immutable storage layers.
Multi-Engine Interoperability: Enable the same table to be queried by Spark, Flink, Trino, Presto, DuckDB, warehouses, and more.

In other words, table formats solve the “wild west of files” problem, turning data lakes into lakehouses that balance scalability with structure. The differences among Iceberg, Delta, Hudi, Paimon, and DuckLake lie in how they achieve this and what trade-offs they make to optimize for batch, streaming, or simplicity.

Next, we’ll walk through the history and evolution of each format to see how these ideas took shape.

The Evolution of Open Table Formats

The journey of open table formats reflects the challenges companies faced as data lakes scaled from terabytes to petabytes. Each format emerged to solve specific pain points:

Apache Hudi (2016) – Created at Uber to solve freshness and incremental ingestion. Hudi pioneered row-level upserts and deletes on data lakes, enabling near real-time pipelines on Hadoop-sized datasets.
Delta Lake (2017–2018) – Developed by Databricks to unify batch and streaming in Spark. Its transaction log design (_delta_log) gave data lakes database-like commits and time-travel capabilities, making it a cornerstone of the “lakehouse” concept.
Apache Iceberg (2018) – Born at Netflix to overcome Hive’s scalability and schema evolution limitations. Its snapshot/manifest-based metadata model provided atomic commits, partition evolution, and reliable time-travel at massive scale, quickly becoming an industry favorite.
Apache Paimon (2022) – Emerging from Alibaba’s Flink ecosystem, Paimon was built streaming-first. Its LSM-tree design optimized for high-throughput upserts and continuous compaction, positioning it as a bridge between real-time CDC ingestion and analytics.
DuckLake (2025) – The newest entrant, introduced by the DuckDB/MotherDuck team. Instead of managing JSON or Avro metadata files, DuckLake stores all table metadata in a relational database. This catalog-centric design aims to simplify consistency, enable multi-table transactions, and drastically speed up query planning.

These formats represent waves of innovation:

First wave (Hudi, Delta): Improving upon the concept of tables on the data lake.
Second wave (Iceberg): focusing on batch reliability, schema evolution, and interoperability.
Third wave (Paimon, DuckLake): rethinking the architecture for real-time data and metadata simplicity.

Next, we’ll dive into Apache Iceberg in detail, its metadata structure, features, and why it has become the default choice for many modern lakehouse deployments.

Apache Iceberg: The Batch-First Powerhouse

Background & Origins
Apache Iceberg was born at Netflix in 2018 and donated to the Apache Software Foundation in 2019. Its mission was clear: fix the long-standing problems of Hive tables, unreliable schema changes, expensive directory scans, and lack of true atomicity. Iceberg introduced a clean-slate design that scaled to petabytes while guaranteeing ACID transactions, schema evolution, and time-travel queries.

Metadata Structure
Iceberg’s metadata model is built on a hierarchy of files:

Table metadata file (JSON): tracks schema versions, partition specs, snapshots, and properties.
Snapshots: each commit creates a new snapshot, representing the table’s full state at that point in time.
Manifest lists & manifests (Avro): hierarchical indexes of data files, enabling partition pruning and column-level stats without scanning entire directories.

This design avoids reliance on directory listings, making planning queries over millions of files feasible.

Core Features

Schema Evolution: Add, drop, or rename columns without breaking queries, thanks to internal column IDs.
Partition Evolution: Change partitioning strategies (e.g., switch from daily to hourly partitions) without rewriting historical data.
Time Travel: Query the table as of a specific snapshot ID or timestamp.
Hidden Partitioning: Abstracts partition logic from users while still enabling efficient pruning.
Optimistic Concurrency: Writers atomically commit new snapshots, with conflict detection to prevent corruption.

Row-Level Changes
Initially copy-on-write, Iceberg now also supports delete files for merge-on-read semantics. Deletes can be tracked separately and applied at read time, reducing write amplification for frequent updates. Background compaction later consolidates these into optimized Parquet files.

Ecosystem & Adoption
Iceberg’s neutrality and technical strengths have driven broad adoption. It is supported in:

Engines: Spark, Flink, Trino, Presto, Hive, Impala, DuckDB.
Cloud platforms: AWS Athena, AWS Glue, Snowflake, BigQuery, Dremio, and more.
Catalogs: Hive Metastore, AWS Glue, Apache Nessie, Polaris.

By late 2024, Iceberg had become the de facto industry standard for open table formats, with adoption by Netflix, Apple, LinkedIn, Adobe, and major cloud vendors. Its community-driven governance and rapid innovation ensure it continues to evolve, recent features like row-level delete vectors and REST catalogs are making it even more capable.

Next, we’ll look at Delta Lake, the transaction-log–driven format that became the backbone of Databricks’ lakehouse vision.

Delta Lake: The Transaction-Log

Background & Origins
Delta Lake was introduced by Databricks around 2017–2018 to address Spark’s biggest gap: reliable transactions on cloud object storage. Open-sourced in 2019 under the Linux Foundation, Delta Lake became the backbone of Databricks’ lakehouse pitch, combining data warehouse reliability with the scalability of data lakes. Its design centered on a simple but powerful idea: use a transaction log to coordinate all changes.

Metadata Structure
At the core of every Delta table is the _delta_log directory:

JSON transaction files: Each commit appends a JSON file describing added/removed data files, schema changes, and table properties.
Checkpoints (Parquet): Periodic checkpoints compact the log for faster reads, storing the authoritative list of active files at a given version.
Versioning: Every commit is versioned sequentially, making time-travel queries straightforward (VERSION AS OF or TIMESTAMP AS OF).

This log-based design is simple and easy to reconstruct: replay JSON logs from the last checkpoint to reach the latest state.

Core Features

ACID Transactions: Ensures consistent reads and writes, even under concurrent Spark jobs.
Schema Enforcement & Evolution: Protects against incompatible writes while allowing schema growth.
Time Travel: Query historical versions for auditing or rollback.
Unified Batch & Streaming: Spark Structured Streaming and batch jobs can read/write the same Delta table, reducing architectural complexity.
Performance Optimizations: Features like Z-order clustering, data skipping, and caching improve query speed (especially in Databricks’ runtime).
Change Data Feed (CDF): Exposes row-level changes between versions, useful for downstream syncs and CDC pipelines.

Row-Level Changes
Delta primarily uses copy-on-write: updates and deletes rewrite entire Parquet files while marking old ones as removed in the log. This guarantees atomicity but can be expensive at scale. To mitigate, Delta introduced deletion vectors (in newer releases), which track row deletions without rewriting whole files, closer to merge-on-read semantics. Upserts are supported via SQL MERGE INTO, commonly used for database change capture workloads.

Ecosystem & Adoption
Delta Lake is strongest in the Spark ecosystem and is the default format in Databricks. It’s also supported by:

Engines: Spark (native), Flink, Trino/Presto (via connectors).
Clouds: AWS EMR, Azure Synapse, and some GCP services.
Libraries: Delta Standalone (Java), Delta Rust, and integrations for Python beyond Spark.

While its openness has improved since Delta 2.0, much of its adoption remains tied to Databricks. Still, Delta Lake is one of the most widely used formats in production, powering pipelines at thousands of organizations.

Next, we’ll explore Apache Hudi, the pioneer of incremental processing and near-real-time data lake ingestion.

Apache Hudi: The Incremental Pioneer

Background & Origins
Apache Hudi (short for Hadoop Upserts Deletes and Incrementals) was created at Uber in 2016 to solve a pressing challenge: keeping Hive tables up to date with fresh, continuously changing data. Uber needed a way to ingest ride updates, user changes, and event streams into their Hadoop data lake without waiting hours for batch jobs. Open-sourced in 2017 and donated to Apache in 2019, Hudi became the first widely adopted table format to support row-level upserts and deletes directly on data lakes.

Metadata Structure
Hudi organizes tables around a commit timeline stored in a .hoodie directory:

Commit files: Metadata describing which data files were added/removed at each commit.
COW vs MOR modes:
- Copy-on-Write (COW): Updates replace entire Parquet files, similar to Iceberg/Delta.
- Merge-on-Read (MOR): Updates land in small Avro delta log files, merged with base Parquet files at read time.
Indexes: Bloom filters or hash indexes help locate records by primary key, making upserts efficient.

This dual-mode design gives engineers control over the trade-off between write latency and read latency.

Core Features

Upserts & Deletes by Key: Guarantees a single latest record per primary key, ideal for CDC ingestion.
Incremental Pulls: Query only the rows changed since a given commit, enabling efficient downstream pipelines.
Compaction: Background jobs merge log files into larger Parquet files for query efficiency.
Savepoints & Rollbacks: Manage table states explicitly, ensuring recovery from bad data loads.
Flexible Indexing: Choose partitioned, global, or custom indexes to balance performance with storage cost.

Row-Level Changes
Hudi was designed for this problem. In COW mode, updates rewrite files. In MOR mode, updates are appended as log blocks, making them queryable almost immediately. Readers can choose:

Snapshot mode (base + logs for freshest data).
Read-optimized mode (compacted base files for speed).

Deletes are handled similarly, either as soft deletes in logs or hard deletes during compaction.

Ecosystem & Adoption
Hudi integrates tightly with:

Engines: Spark (native datasource), Flink (growing support), Hive, Trino/Presto.
Clouds: AWS EMR and AWS Glue have built-in Hudi support, making it popular on S3.
Streaming: Confluent Kafka, Debezium, and Flink CDC can stream directly into Hudi tables.

While Iceberg and Delta now dominate conversations, Hudi remains a strong choice for near real-time ingestion and CDC use cases, particularly in AWS-centric stacks. Its flexibility (COW vs MOR) and incremental consumption features make it especially valuable for pipelines that need fast data freshness without sacrificing reliability.

Next, we’ll examine Apache Paimon, the streaming-first format that extends Hudi’s incremental vision with an LSM-tree architecture.

Apache Paimon: Streaming-First by Design

Background & Origins
Apache Paimon began life as Flink Table Store at Alibaba in 2022, targeting the need for continuous, real-time data ingestion directly into data lakes. It entered the Apache Incubator in 2023 under the name Paimon. Unlike Iceberg or Delta, which started with batch analytics and later added streaming features, Paimon was streaming-first. Its mission: make data lakes act like a materialized view that is always up to date.

Metadata & Architecture
Paimon uses a Log-Structured Merge-tree (LSM) design inspired by database internals:

MemTables and flushes: Incoming data is written to in-memory buffers, then flushed to small immutable files.
Multi-level compaction: Files are continuously merged into larger sorted files in the background.
Snapshots: Each compaction or commit produces a new snapshot, allowing both batch queries and streaming reads.
Primary-key awareness: Tables can enforce keys and apply merge rules (e.g., last-write-wins or aggregate merges).

This architecture makes frequent row-level changes cheap (append-only writes) while deferring heavy merges to compaction tasks.

Core Features

Real-Time Upserts & Deletes: Native support for continuous CDC ingestion with efficient row-level operations.
Merge Engines: Configurable rules for handling key collisions (e.g., overwrite, aggregate, or log-append).
Dual Read Modes: Query as a static snapshot (batch) or as a change stream (streaming).
Streaming/Batch Unification: The same table can power batch analytics and real-time dashboards.
Deletion Vectors: Efficiently tracks row deletions without rewriting base files.

Row-Level Changes
Unlike Iceberg (COW with delete files) or Delta (COW with deletion vectors), Paimon is natively merge-on-read. Updates and deletes are appended as small log segments, queryable immediately. Background compaction gradually merges them into optimized columnar files. This makes Paimon highly efficient for high-velocity workloads like IoT streams, CDC pipelines, or real-time leaderboards.

Ecosystem & Adoption
Paimon integrates tightly with Apache Flink, where it feels like a natural extension of Flink SQL. It also has growing support for Spark, Hive, Trino/Presto, and OLAP systems like StarRocks and Doris. Adoption is strongest among teams building streaming lakehouses, particularly those already invested in Flink. While younger than Iceberg or Delta, Paimon is rapidly attracting attention as organizations push for sub-minute data freshness.

Next, we’ll turn to DuckLake, the newest entrant that rethinks table metadata management by moving it entirely into SQL databases.

DuckLake: Metadata Reimagined with SQL

Background & Origins
DuckLake is the newest table format, introduced in 2025 by the DuckDB and MotherDuck teams. Unlike earlier formats that manage metadata with JSON logs or Avro manifests, DuckLake flips the script: it stores all table metadata in a relational SQL database. This approach is inspired by how cloud warehouses like Snowflake and BigQuery already manage metadata internally, but DuckLake makes it open and interoperable.

Metadata & Architecture

SQL Catalog: Metadata such as snapshots, schemas, file lists, and statistics are persisted as ordinary relational tables.
Transactions: Updates to metadata happen through standard SQL transactions, ensuring strong ACID guarantees without relying on object-store semantics.
Multi-Table Transactions: Because it’s database-backed, DuckLake supports atomic operations across multiple tables, something file-based formats struggle with.
File Storage: Data remains in Parquet files on cloud or local storage, DuckLake just replaces the metadata layer with SQL.

This design dramatically reduces the complexity of planning queries (no manifest scanning), makes commits faster, and enables features like cross-table consistency (possible in Apache Iceberg if using the Nessie catalog).

Core Features

SQL-Native Metadata: Easy to query, debug, or extend using plain SQL.
Fast Commits & Planning: Small updates don’t require writing multiple manifest files, just SQL inserts/updates.
Cross-Table Atomicity: Multi-table changes commit together, a unique strength.
Familiar Deployment: The catalog can run on DuckDB, PostgreSQL, or any transactional SQL database.

Row-Level Changes
DuckLake handles updates and deletes via copy-on-write on Parquet files, but the metadata transaction is nearly instantaneous. Row-level changes are coordinated by the SQL catalog, avoiding the latency and eventual consistency pitfalls of cloud storage–based logs. In effect, DuckLake behaves like Iceberg for data files but with much faster commit cycles.

Ecosystem & Adoption

Primary Engine: DuckDB, via a DuckLake extension.
Potential Integrations: Any SQL-aware engine could adopt DuckLake, since the catalog is just relational tables.
Use Cases: Analytics sandboxes, developer-friendly data apps, and teams seeking simplicity without deploying heavy metadata services.

As of 2025, DuckLake is still young but has sparked excitement by simplifying lakehouse architecture. It’s best seen as a complement to more mature formats, with particular appeal to DuckDB users and teams tired of managing complex metadata stacks.

Next, we’ll step back and compare all five formats side by side, looking at metadata design, row-level update strategies, ecosystem support, and adoption trends.

Comparing the Open Table Formats

Now that we’ve walked through each format individually, let’s compare them across the dimensions that matter most to data engineers and architects.

1. Metadata Architecture

Iceberg: Hierarchical snapshots + manifests. Excellent for pruning large datasets but metadata can be complex.
Delta Lake: Sequential transaction log (_delta_log). Simple and efficient for versioning, but logs can grow large without checkpoints.
Hudi: Commit timeline with optional delta logs. Flexible but more operational overhead.
Paimon: LSM-tree style compaction with snapshots. Streaming-friendly and highly write-efficient.
DuckLake: Metadata in a SQL database. Simplifies commits and query planning, enables multi-table transactions.

2. Row-Level Changes

Iceberg: Copy-on-write by default, with delete files for merge-on-read.
Delta Lake: Copy-on-write, plus deletion vectors in newer versions.
Hudi: Dual modes: COW for read-optimized, MOR for low-latency upserts.
Paimon: Always merge-on-read via LSM-tree segments, optimized for frequent updates.
DuckLake: Copy-on-write, but with faster commit cycles thanks to SQL-backed metadata.

3. Ecosystem Support

Iceberg: Widest engine support (Spark, Flink, Trino, Presto, Hive, Snowflake, Athena, BigQuery, Dremio, DuckDB).
Delta Lake: Deep Spark and Databricks integration; expanding connectors for Flink and Trino.
Hudi: Strong in Spark, Hive, Presto, and AWS (Glue, EMR). Flink support growing.
Paimon: Native to Flink; Spark and Trino integration improving; also ties to OLAP systems like Doris/StarRocks.
DuckLake: Early-stage, centered on DuckDB; potential for other SQL engines to adopt.

4. Adoption Trends

Iceberg: Emerging as the industry standard for open table formats, with broad vendor alignment.
Delta Lake: Dominant within Databricks/Spark ecosystems; adoption tied to Databricks customers.
Hudi: Niche but strong in CDC and near real-time use cases; proven at scale in companies like Uber.
Paimon: Rising fast in the Flink/streaming community; positioned as the “streaming lakehouse” format.
DuckLake: Newest entrant, appealing for simplicity and developer-friendliness; adoption still experimental.

Next, we’ll step back and examine industry trends shaping the adoption of these formats and what they signal for the future of the lakehouse ecosystem.

Industry Trends in Table Format Adoption

The “table format wars” of the past few years are starting to settle into clear patterns of adoption. While no single format dominates every use case, the industry is coalescing around certain choices based on scale, latency, and ecosystem needs.

Iceberg as the Default Standard
Iceberg has emerged as the most widely supported and vendor-neutral choice. Cloud providers like AWS, Google, and Snowflake have all added native support, and query engines like Trino, Presto, Hive, and Flink integrate with it out-of-the-box. Its Apache governance and cross-engine compatibility make it the safe long-term bet for enterprises standardizing on a single open format.

Delta Lake in the Spark/Databricks World
Delta Lake remains the default in Spark- and Databricks-heavy shops. Its simplicity (transaction logs) and seamless batch/stream integration continue to attract teams already invested in Spark. While its ecosystem is narrower than Iceberg’s, Delta Lake’s deep integration with Databricks runtime and machine learning workflows ensures strong adoption in that ecosystem.

Hudi in CDC and Incremental Ingestion
Hudi carved out a niche in change data capture (CDC) and near real-time ingestion. Telecom, fintech, and e-commerce companies still rely on Hudi for incremental pipelines, especially on AWS where Glue and EMR make it easy to deploy. While Iceberg and Delta have added incremental features, Hudi’s head start and MOR tables keep it relevant for low-latency ingestion scenarios.

Paimon and the Rise of Streaming Lakehouses
As real-time analytics demand grows, Paimon is gaining momentum in the Flink community and among companies building streaming-first pipelines. Its LSM-tree design positions it as the go-to choice for high-velocity data, IoT streams, and CDC-heavy architectures. Although young, its momentum signals a broader shift: the next wave of lakehouse innovation is about sub-minute freshness.

DuckLake and Metadata Simplification
DuckLake reflects a newer trend: rethinking metadata management. By moving metadata into SQL databases, it dramatically simplifies operations and enables cross-table transactions. Adoption is still experimental, but DuckLake has sparked interest among teams who want lakehouse features without managing complex catalogs or metastores. Its trajectory will likely influence how future formats handle metadata.

Convergence and Interoperability
One notable trend: features are converging. Iceberg now supports row-level deletes via delete files; Delta added deletion vectors; Hudi and Paimon both emphasize streaming upserts. Tooling is also evolving toward interoperability, catalog services like Apache Nessie and Polaris aim to support multiple formats, and BI engines increasingly connect to all.

In short:

Iceberg is becoming the industry’s lingua franca.
Delta thrives in Databricks-first stacks.
Hudi holds ground in CDC and incremental ingestion.
Paimon is rising with real-time streaming needs.
DuckLake challenges conventions with SQL-backed simplicity.

Next, we’ll wrap up with guidance on how to choose the right format based on your workloads, ecosystem, and data engineering priorities.

Choosing the Right Open Table Format

With five strong options on the table, Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, the choice depends less on “which is best” and more on which aligns with your workloads, ecosystem, and priorities. Here’s how to think about it:

When to Choose Apache Iceberg

You want the broadest engine and vendor support (Spark, Flink, Trino, Presto, Hive, Dremio, Snowflake, BigQuery, etc.).
Your workloads are batch-heavy and prioritize consistent snapshots, schema evolution, and large-scale analytics.
You want to standardize on the emerging industry default with the widest community and neutral Apache governance.

When to Choose Delta Lake

Your data stack is Databricks-first or heavily Spark-centric.
You need seamless batch + streaming unification with Spark Structured Streaming.
You value Databricks’ ecosystem of optimizations (e.g., Z-order, caching, machine learning integrations).

When to Choose Apache Hudi

You need frequent upserts and deletes on data lakes.
Your pipelines depend on incremental consumption of data (only new/changed rows since the last commit).
You want a proven option for CDC ingestion and near real-time pipelines, especially on AWS Glue/EMR.

When to Choose Apache Paimon

Your workloads are streaming-first, with high-velocity CDC or IoT data.
You want to unify real-time and batch processing within the same table.
You’re already invested in Apache Flink and want a table format purpose-built for it.

When to Choose DuckLake

You want simplicity in metadata management (SQL instead of JSON/Avro manifests).
You’re working in DuckDB/MotherDuck environments or need lightweight lakehouse capabilities.
You value fast commits, easy debugging, and multi-table atomicity, even if the format is newer and less battle-tested.

Final Takeaway

Iceberg: the universal standard for long-term interoperability.
Delta: the Databricks/Spark-native option.
Hudi: the incremental/CDC pioneer.
Paimon: the streaming-first disruptor.
DuckLake: the metadata simplifier.

No matter which you choose, adopting an open table format is the key to turning your data lake into a true lakehouse: reliable, flexible, and future-proof.

Conclusion

Open table formats are no longer niche, they’re the foundation of the modern data stack. Whether your challenge is batch analytics, real-time ingestion, or simplifying metadata, there’s a format designed to meet your needs. The smart path forward isn’t just picking one blindly, but aligning your choice with your data velocity, tooling ecosystem, and long-term governance strategy.

In practice, many organizations run more than one format side by side. The good news: as open standards mature, interoperability and ecosystem support are expanding, making it easier to evolve over time without locking yourself into a dead end.

The lakehouse era is here, and open table formats are its backbone.

Get Data Lakehouse Books:

Join the Data Lakehouse Community Data Lakehouse Blog Roll

The 2025 & 2026 Ultimate Guide to the Data Lakehouse and the Data Lakehouse Ecosystem

Alex Merced — Tue, 23 Sep 2025 09:00:00 GMT

Year-end 2025 reflections, looking ahead to 2026

Over the past few years, data platforms have crossed a tipping point. Rigid, centralized warehouses proved great at trustworthy BI but struggled with diverse data and elastic scale. Open data lakes delivered low-cost storage and freedom of choice, yet lacked the transactional rigor and performance guarantees analytics teams rely on. In 2025, the data lakehouse matured from an idea into an operating model: open table formats, transactional metadata, and multi-engine access over a single, governed body of data.

This guide distills what changed, why it matters, and how to put it to work. We’ll start by clarifying where warehouses shine and crack, where lakes empower and swamp, and why older directory-based table designs (think classic Hive tables) hit scaling and consistency limits. From there, we’ll show how modern table formats, Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon, solved those limits by tracking files and snapshots instead of folders, enabling ACID transactions, time travel, and intelligent pruning at petabyte scale.

2025 also cemented a practical reference architecture. A successful lakehouse now looks less like a monolith and more like a layered system: cloud object storage for durability and cost, an open table format for transactions and evolution, ingestion that blends batch and streaming, a catalog for governance and discoverability, and a flexible consumption layer that serves SQL, BI, notebooks, and AI agents with consistent semantics.

Why now? Three forces converged this year:

Streaming-by-default workloads turned “daily batch” into “continuous micro-batch,” demanding exactly-once commits and small-file management.
AI and agentic workflows moved from proofs of concept to production, generating highly variable, ad-hoc queries that require low-latency acceleration without brittle hand-tuning.
Open interoperability became table stakes, organizations want one source of truth read by many engines, not many copies of truth managed by many teams.

This guide is an accessible deep dive for Data Engineers and Data Architects. You’ll get a clear mental model of the formats, their metadata structures, and the operational playbook: compaction, snapshot expiration, partition evolution, and reflection/materialization strategies for speed at scale. We’ll also survey the ingestion and streaming ecosystem (connectors, CDC, stream processors), Python-native options for lakehouse workloads (Polars, DuckDB, DataFusion, Daft, Dask), and emerging edge patterns where inference runs close to the data.

Finally, we’ll close with a curated reading list, books and long-form resources that stood out in 2025, and pragmatic guidance on choosing components in 2026 without locking yourself in. If your mandate is to deliver trustworthy, performant, and AI-ready analytics on open data, this guide is your map.

The Challenges in Modern Data Architecture

The rise of the lakehouse didn’t happen in a vacuum. It emerged as a response to the very real challenges of yesterday’s dominant architectures, data warehouses and data lakes. Understanding their strengths and weaknesses sets the stage for why the lakehouse model became inevitable.

Data Warehouses: Strength in Structure, Weakness in Flexibility

Data warehouses provided the first true enterprise-scale analytics platforms. They enforced schema-on-write, ensuring data quality and making business intelligence consistent across the organization. For years, this was invaluable: clean, curated, trusted dashboards.

But the cracks widened in the 2010s and 2020s:

Rigid schemas: Every change to a source system meant heavy ETL work to keep the warehouse schema in sync. New data types, JSON, images, sensor streams, didn’t fit neatly into tables.
High costs: Warehouses couple compute and storage. Scaling for more data or users often meant overpaying for resources you didn’t fully use.
Latency in data freshness: The ETL pipelines that fed warehouses ran daily or hourly, leaving decision-makers working with stale data.
Limited AI/ML support: Warehouses excel at structured SQL queries but aren’t designed to handle the diverse, unstructured, and large-scale data needed for machine learning.

Warehouses solved consistency but at the price of agility.

Data Lakes: Flexibility Meets the “Data Swamp”

Enter data lakes. By shifting to schema-on-read, organizations gained the freedom to store anything: logs, media, documents, semi-structured JSON, raw database dumps. Storage costs plummeted thanks to cloud object stores like S3 and ADLS, and data scientists loved having raw, unmodeled data at their fingertips.

But flexibility introduced new pain:

Data quality issues: With no enforced schema, data lakes quickly devolved into “data swamps”, vast, uncurated collections of files that few trusted.
Poor governance: Security, lineage, and access controls were bolted on, often inconsistently. Teams struggled to know what data existed and whether it was safe to use.
Performance bottlenecks: Query engines like Hive, Spark, or Presto had to scan massive directories of files. Without transactional guarantees, concurrent writes could corrupt datasets or leave analysts with incomplete results.
High operational overhead: Managing partitions, small files, and manual compactions became part of daily operations.

Lakes solved agility but at the price of trust.

The gap was clear: warehouses offered trust but no flexibility, while lakes offered flexibility but no trust. By the early 2020s, organizations wanted the best of both, structured reliability and open flexibility, laying the foundation for the modern data lakehouse.

What Is Hive and the Challenges of Hive Tables

Before the lakehouse era, Apache Hive was the workhorse that made large-scale data in Hadoop clusters queryable with SQL. Hive introduced the Hive Metastore, which stored table definitions (schemas, partitions, and locations), enabling analysts to run SQL-like queries over files sitting in HDFS or cloud storage. It was one of the first major attempts to give a data-lake-like environment a relational feel.

But Hive’s approach, tracking directories of files as tables, brought structural limitations that became bottlenecks as datasets and expectations grew.

Directory-Centric Table Management

In Hive, each table maps to a folder, and each partition to a subfolder. Query engines scan these directories at runtime to discover files. While this worked when data volumes were modest, modern cloud object stores made directory scans painfully slow. Listing millions of files before executing a query often dominated total query time.

Lack of ACID Transactions

Hive tables were essentially append-only. Without built-in transactions, concurrent writers risked corrupting tables, and readers could encounter partial data during an update. Later ACID extensions attempted to patch this with delta files and compaction, but these added complexity and overhead, and weren’t consistently supported across engines.

Painful Updates and Schema Evolution

Modifying data in Hive tables was inefficient:

Updates and deletes required rewriting entire partitions or entire tables.
Schema changes (like renaming a column) often broke downstream jobs or forced costly rewrites.

The result: rigid datasets that were expensive to maintain and slow to evolve with business needs.

The Small Files Problem

Hive ingestion pipelines, especially those running frequently, created floods of small files. Query performance degraded because engines had to open and read from thousands of tiny files. Without built-in small-file management, engineers had to implement periodic compaction jobs to maintain performance.

Hive was a critical stepping stone: it proved the value of SQL on big data and inspired the metadata-driven approach all lakehouse formats now follow. But its reliance on directory-based tracking and limited support for transactional, evolving workloads ultimately constrained its ability to power the next generation of data platforms.

The Innovation of Tracking Tables by Tracking Files vs. Tracking Directories

The turning point from Hive-style tables to modern lakehouse formats came with a deceptively simple idea:
stop tracking directories of files, and start tracking individual files in metadata.

Why Directories Fall Short

Directory-based tracking (as in Hive) meant that the engine had to:

List every file in a partition directory before running a query.
Infer table state from the file system at query time.

This created major problems in cloud storage, where operations like LIST are slow and expensive. It also made concurrency hard, two jobs writing to the same folder could overwrite each other’s files without the catalog knowing until it was too late.

File-Level Tracking

Modern table formats introduced file-level manifests: structured metadata that explicitly records every file that belongs to a table, along with statistics about its contents. Instead of scanning folders, engines read this compact metadata to know exactly which files to use.

Benefits include:

Faster planning: Queries skip expensive directory listings, instead reading a few manifest files that describe thousands of data files.
Atomic commits: Updates create a new manifest (or snapshot) in a single operation. Readers either see the old version or the new one, never a half-written state.
Schema evolution: Metadata can track multiple schema versions, allowing columns to be added, renamed, or dropped without rewriting entire datasets.
Partition independence: Partitioning is recorded in metadata, not folder structures, enabling hidden partitions and even partition evolution over time.
Fine-grained deletes and upserts: Since every file is individually tracked, formats can support row-level operations by marking old files as deleted and adding new ones.

Snapshots and Time Travel

By treating metadata itself as a versioned object, table formats unlocked time travel: the ability to query data as it existed at any point in time. Each snapshot references a specific set of files, creating a complete, immutable view of the table at that moment.

This shift, from directories to files, from implicit state to explicit metadata, transformed raw data lakes into reliable, database-like systems. It’s the foundation that made the lakehouse architecture possible and paved the way for the new generation of table formats.

The New Generation of Data Lake Tables: Iceberg, Delta, Hudi, and Paimon

With file-level tracking as the breakthrough, several open-source projects emerged to redefine how data lakes operate. These table formats provide the transactional, metadata-rich foundation that transforms a raw data lake into a full-fledged lakehouse. Each project shares core principles, ACID transactions, schema evolution, and time travel, but emphasizes different strengths.

Apache Iceberg

Born at Netflix and now a top-level Apache project, Iceberg is designed for engine-agnostic interoperability. Its hierarchical metadata structure (table metadata → manifest lists → manifest files) allows scaling to billions of files while still enabling fast query planning.
Key features:

Hidden partitioning and partition evolution.
Broad engine support (Spark, Flink, Trino, Presto, Dremio, and more).
Strong focus on openness through the REST catalog API.
Rich schema evolution, including column renames and type promotions.

Iceberg has become a de facto standard for enterprises seeking an open, future-proof lakehouse.

Delta Lake

Originally created by Databricks, Delta Lake popularized the concept of a transactional log for data lakes. It uses an append-only transaction log (_delta_log) with JSON entries and Parquet checkpoints to track file state.
Key features:

ACID transactions tightly integrated with Apache Spark.
Time travel and schema evolution.
Optimizations like Z-Ordering for clustering.
Deep integration with the Databricks ecosystem, though community adoption beyond Spark is growing.

Delta remains particularly strong for teams standardized on Databricks or Spark-centric workflows.

Apache Hudi

Developed at Uber, Hudi was one of the earliest attempts to bring database-like capabilities to data lakes. It excels at incremental processing and change data capture (CDC).
Key features:

Two storage modes: Copy-on-Write (CoW) for read-optimized workloads, and Merge-on-Read (MoR) for write-heavy, near-real-time use cases.
Native upserts and deletes.
Built-in indexing for record-level operations.
Tight integrations with Spark, Flink, and Hive.

Hudi is especially attractive for pipelines that demand frequent updates and streaming ingestion.

Apache Paimon

A newer entrant, Paimon (formerly Flink Table Store) emphasizes streaming-first lakehouse design. It uses an LSM-tree style file organization to unify batch and stream processing.
Key features:

Native CDC and incremental queries.
Deep integration with Apache Flink.
Snapshot isolation with continuous compaction.
Growing ecosystem to support Spark, Hive, and beyond.

Paimon fills a niche where real-time data ingestion and analytics converge, making it compelling for event-driven architectures.

Together, these formats represent the evolution from directory-based tables to transactional, metadata-driven lakehouse systems. Each brings a unique philosophy: Iceberg for openness, Delta for Spark-native simplicity, Hudi for streaming updates, and Paimon for unified batch-stream processing. Understanding their trade-offs is critical when designing a modern data platform.

Fundamental Architecture of the Data Lakehouse

At its core, the data lakehouse is not a single product but an architectural pattern. It blends the scalability and openness of data lakes with the transactional reliability and governance of data warehouses. By 2025, a consensus emerged: a lakehouse succeeds when it clearly defines layers, each with its own role but working together as a cohesive whole.

Storage as the Foundation

The lakehouse begins with cloud object storage (e.g., Amazon S3, Azure Data Lake Storage, Google Cloud Storage). This layer offers low-cost, durable, infinitely scalable storage for all file types. Unlike warehouses, it decouples compute from storage, multiple engines can read from the same data without duplicating it.

Metadata and Table Formats

On top of storage sits a table format, the metadata layer that turns a set of files into a logical table. Formats like Iceberg, Delta, Hudi, and Paimon bring:

ACID transactions
Schema enforcement and evolution
Partition pruning and statistics for efficient queries
Time travel through snapshot-based metadata

This layer is what transforms a “data swamp” into structured, queryable datasets.

Catalog and Governance

A catalog connects metadata to the outside world. It tracks what tables exist and their locations, while enforcing access policies and governance rules. Think of it as the bridge between storage and consumption. Examples include Hive Metastore, AWS Glue, Unity Catalog, Dremio Catalog and open-source options like Nessie or Apache Polaris.

Compute and Federation

Query engines like Dremio, Trino, Spark, and Flink sit on top, accessing tables via the catalog. These engines provide federation, joining and querying data from multiple systems, and execute transformations, BI queries, or machine learning pipelines. The lakehouse architecture allows multiple engines to share the same data without conflict.

Consumption and Semantics

Finally, end users connect through BI dashboards, notebooks, or AI systems. A semantic layer often sits here, defining consistent metrics and business concepts across tools. This ensures a “single version of truth” for everyone consuming data.

This layered design, storage, table format, catalog, compute, and consumption, has become the reference architecture for the modern data lakehouse. It solves the warehouse vs. lake tradeoff by delivering flexibility, trust, and performance in one unified stack.

Metadata Structures Across Modern Table Formats

While all lakehouse table formats share the principle of tracking files rather than directories, each one implements its own metadata architecture. Understanding these differences is crucial for choosing the right format and for operating them at scale.

Apache Iceberg

Snapshots: Every commit creates a new snapshot that references a set of files.
Manifests: Each snapshot points to manifest lists, which then point to manifest files. These manifest files contain the actual list of data files, along with stats like min/max values for columns.
Table Metadata File: A JSON/Avro file storing schema versions, partition specs, snapshot history, and pointers to the current snapshot.
Strengths: Hierarchical design scales to billions of files, supports hidden partitioning, and makes time travel lightweight.

Delta Lake

Transaction Log: All operations are recorded in an append-only _delta_log directory as JSON files.
Checkpoints: Periodically, the log is compacted into Parquet checkpoint files for faster reads.
Table State: Current table state is reconstructed by combining the latest checkpoint with newer JSON entries.
Strengths: Simple, linear log model tightly integrated with Spark; efficient for workloads within the Databricks ecosystem.

Apache Hudi

Timeline: A series of commit, deltacommit, and compaction files in a .hoodie directory describe changes.
Storage Modes:
- Copy-on-Write (CoW): rewrites files on update.
- Merge-on-Read (MoR): writes delta logs and later compacts them with base files.
Indexing: Optional record-level indexes accelerate upserts and deletes.
Strengths: Optimized for streaming ingestion and CDC use cases, with incremental pull queries.

Apache Paimon

LSM-Tree Inspired: Uses log segments and compaction levels, optimized for high-frequency updates.
Snapshots: Metadata tracks current file sets and supports branching for consistent queries.
Changelog Streams: Natively emits row-level changes for downstream streaming consumers.
Strengths: Built for unified batch and streaming, with strong Flink integration.

In summary:

Iceberg emphasizes scalability and cross-engine interoperability.
Delta Lake focuses on simplicity and Spark-native performance.
Hudi delivers real-time upserts and incremental views.
Paimon pioneers streaming-first design with changelogs.

These metadata designs reflect each project’s philosophy, and they form the backbone of how the modern lakehouse balances flexibility, consistency, and speed.

Implementing a Lakehouse: The Five Core Layers

Designing a modern lakehouse isn’t about choosing a single tool, it’s about assembling the right components across five architectural layers. Each layer has its own responsibilities, and together they create a system that is scalable, governed, and usable for analytics and AI.

Deep Dive into the 5 layers is the core of the book "Architecting an Apache Iceberg Lakehouse"

1. Storage Layer

This is the foundation: low-cost, durable storage capable of holding structured, semi-structured, and unstructured data.

Common choices: Amazon S3, Azure Data Lake Storage, Google Cloud Storage, or on-prem HDFS/MinIO.
Data is stored in open formats such as Parquet, ORC, or Avro.
Separation of storage from compute allows multiple engines to share the same data without duplication.

2. Table Format Layer

Here, the metadata format gives structure and reliability to raw files.

Options: Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon.
Capabilities include ACID transactions, schema evolution, partition pruning, and time travel.
This layer transforms the lake from a “data swamp” into a transactional system of record.

3. Ingestion Layer

The ingestion layer handles data movement into the lakehouse.

Batch ingestion: Tools like Fivetran, Airbyte, Estuary, Hevo or custom ETL jobs land data periodically.
Streaming ingestion: Systems like Confluent, Aiven, StreamNative, RisingWave, Kafka, Redpanda, Pulsar, or Flink push events into table formats in near real-time.
Goal: balance freshness, cost, and reliability while avoiding problems like excessive small files.

4. Catalog & Governance Layer

The catalog is the central registry of your tables, schemas, and access rules.

Examples: Hive Metastore, AWS Glue, Unity Catalog, Dremio Catalog, open-source catalogs like Nessie or Apache Polaris.
Responsibilities: discovery, schema validation, access control, lineage, and auditability.
Acts as the bridge between storage and compute, ensuring data is both secure and discoverable.

5. Federation & Consumption Layer

At the top, query engines and semantic layers make data consumable.

Query federation engines like Dremio or Trino can join lakehouse tables with other sources.
Consumption tools include BI platforms (Tableau, Power BI, Looker), notebooks, and AI agents.
A semantic layer ensures consistency by defining metrics and business terms across all tools.

In practice: these five layers form the blueprint of every successful lakehouse. They separate concerns, storage, metadata, movement, governance, and consumption, while enabling interoperability. The result is a unified platform that scales with data growth, adapts to new workloads, and keeps analytics both flexible and trustworthy.

Lakehouse Ingestion

Once the foundational layers are in place, the next challenge is getting data into the lakehouse efficiently and reliably. Ingestion strategies determine not only data freshness but also table health, file organization, and downstream usability.

Batch Ingestion

Batch remains the most common entry point:

ETL/ELT Services: Tools like Fivetran and Airbyte extract data from SaaS applications, relational databases, and APIs, then land it in cloud object storage. Many now write directly into open table formats (Iceberg, Delta, Hudi) rather than dumping raw CSVs.
Custom Jobs: Python, Spark, or dbt pipelines often transform and load data on schedules, nightly, hourly, or in micro-batches.
Advantages: Predictable loads, simpler monitoring, and often easier cost control.
Challenges: Data freshness is limited by schedule, and frequent batches can generate lots of small files if not managed.

Streaming Ingestion

Real-time data is no longer a luxury, it’s an expectation in 2026:

Event Streams: Platforms like Apache Kafka, Redpanda, Aiven, Confluent, RisingWave, StreamNative, and Apache Pulsar capture streams of events (e.g., clickstream, IoT data) and push them into the lakehouse using connectors or stream processors.
CDC Pipelines: Change data capture tools (Debezium, Estuary Flow) replicate updates from operational databases into Iceberg or Delta tables with low latency.
Stream Processing Engines: Apache Flink and Spark Structured Streaming can apply transformations inline, then commit results directly to lakehouse tables.

Small File Management

One critical concern in ingestion is avoiding a small files problem:

Each tiny file adds overhead to query planning.
Solutions include writer-side batching, file-size thresholds, and downstream compaction jobs.
Modern ingestion platforms often integrate with the table format’s APIs to commit larger, optimized files.

Reliability and Governance

Ingestion isn’t just about moving bytes, it’s about ensuring trustworthy pipelines:

Idempotency: Re-runs shouldn’t create duplicates.
Schema Drift Handling: New source columns should be gracefully added to lakehouse tables with metadata updates.
Monitoring: Data observability platforms (Monte Carlo, Bigeye) can alert when loads fail or data volumes deviate unexpectedly.

In short, ingestion is where data quality meets data freshness. A strong strategy combines batch tools for breadth (ingesting from many SaaS and DB sources) with streaming pipelines for depth (real-time operational data), all while keeping file sizes healthy and metadata consistent.

Lakehouse Streaming

If 2025 was the year of “batch meets real-time,” then 2026 is the year of streaming-first lakehouses. Instead of treating streaming as an afterthought, the modern lakehouse expects ingestion, processing, and query serving to happen continuously. This shift is powered by both table format features (incremental commits, changelogs) and by the streaming ecosystem maturing around open lakehouse standards.

Confluent

As the commercial steward of Apache Kafka, Confluent has led in making streams and tables converge. Their Tableflow and Stream Designer products now write directly to Iceberg and Delta Lake, providing exactly-once guarantees and seamless CDC ingestion. This reduces the need for custom Flink or Spark jobs, Kafka topics become queryable lakehouse tables in real time.

Aiven

Aiven, a managed open-source data platform provider, has expanded its Kafka, Flink, and Postgres services with native Iceberg integrations. Their goal: give teams a turnkey way to capture events, run stream transformations, and land results directly into a governed lakehouse, without stitching together multiple vendors.

Redpanda

Redpanda brings Kafka-API-compatible streaming with higher throughput and lower latency, and in 2025 introduced Iceberg Topics. With this feature, every topic can materialize into an Iceberg table automatically, combining log storage with table metadata. This means developers can treat the same data as both a stream and a table, depending on the workload.

StreamNative

Built around Apache Pulsar, StreamNative pushes the lakehouse deeper into event-driven architectures. Pulsar’s tiered storage, combined with integrations for Iceberg and Delta, means historical message backlogs can be instantly queryable as tables. Their work on unifying messaging and lakehouse storage blurs the boundary between stream broker and data platform.

RisingWave

RisingWave focuses on streaming databases: continuously maintaining materialized views over streams. Its integration with Iceberg allows those real-time views to be published directly into the lakehouse, governed alongside batch data. This bridges operational analytics (e.g., monitoring metrics in near real time) with historical analytics in the same architecture.

Other Notables

Materialize: A streaming database that outputs real-time materialized views, often targeting data lakes and warehouses as sinks.
ksqlDB: Kafka-native SQL for defining streaming transformations, which can also materialize tables into downstream lakehouse storage.
Apache Flink: Still the backbone of many custom streaming-to-lakehouse pipelines, powering advanced transformations before committing results to Iceberg, Hudi, or Delta.

Takeaway: Streaming is no longer bolted onto the lakehouse, it is embedded. Whether through Kafka, Redpanda, Pulsar, Flink, or streaming databases like RisingWave and Materialize, streams now flow directly into transactional tables. The result is a lakehouse where batch and real-time are not two separate worlds but a single, unified system delivering always-fresh data.

Lakehouse Catalogs: Architecture, Compatibility & When to Use Which

A lakehouse catalog is the control plane for your open tables, it tracks metadata locations, permissions, and exposes standard APIs to every engine. Below is a concise, practitioner-focused map of today’s major options and how they fit into a multi-engine, multi-cloud lakehouse.

Apache Polaris (Incubating)

What it is: An open-source, fully featured Apache Iceberg REST catalog designed for vendor-neutral, multi-engine interoperability. Backed by multiple vendors and born from a cross-industry push to standardize the Iceberg catalog layer.
Where it shines: Teams standardizing on Iceberg who want a portable, community-governed catalog that Spark, Flink, Trino, Dremio, StarRocks/Doris can all use via the Iceberg REST API.
Notable: Open governance and REST-by-default avoid lock-in and simplify multi-engine access. Also has the feature to federate other catalogs and soon other table sources.

Apache Gravitino (Incubating)

What it is: A geo-distributed, federated metadata lake that manages metadata in place across heterogeneous sources (file stores, RDBMS, streams) and exposes a unified view to engines like Spark/Trino/Flink. Where it shines: Hybrid/multi-cloud estates with multiple catalogs and sources that need one governance and discovery layer without migrations. Notable: “Catalog of catalogs” approach; can present Iceberg/Hive/Paimon/Hudi catalogs under one umbrella.

AWS Glue Data Catalog (+ Lake Formation)

What it is: AWS’s managed Hive-compatible catalog with first-party governance (Lake Formation) and native support for Iceberg/Delta/Hudi tables in S3, consumed by Athena, EMR, Redshift Spectrum, and Glue jobs. Where it shines: All-in AWS lakehouses needing centralized metadata and fine-grained access control enforced across AWS analytics services. Notable: Managed, integrated, and convenient, cloud-specific by design.

Microsoft OneLake Catalog (Fabric)

What it is: The central catalog for Microsoft Fabric’s “OneLake”, a tenant-wide, Delta-native lake with unified discovery (“Explore”) and governance (“Govern”) experiences. Where it shines: Fabric-centric stacks that want a single catalog for Spark, SQL, Power BI, and Real-Time Analytics over Delta tables in ADLS/OneLake. Notable: Deeply integrated SaaS experience; shortcuts/mirroring help connect external sources, but it’s Azure/Fabric-scoped.

Google BigLake (Metastore + Iceberg)

What it is: Google’s open lakehouse layer: BigLake Metastore catalogs Iceberg tables on GCS; BigQuery reads them natively while Spark/Flink and other engines use the Iceberg REST interface. Where it shines: GCP stacks wanting warehouse-grade operations (BigQuery) over open Iceberg tables stored in customer buckets with multi-engine access. Notable: Managed table maintenance and unified governance via Dataplex/BigLake; Iceberg-first approach.

Project Nessie

What it is: A Git-like, transactional catalog for data lakes that adds branches, tags, time-travel, and cross-table commits on top of Iceberg. Where it shines: Teams needing dev/test isolation, reproducibility, or multi-table atomic commits in an Iceberg lakehouse. Notable: Works with Spark, Flink, Trino, Dremio; deploy anywhere (K8s/containers). Complements standard catalogs with versioning semantics.

Unity Catalog (Open Source)

What it is: An open-sourced universal catalog for data & AI assets with multi-format (Delta, Iceberg via REST/UniForm, files) and multi-engine ambitions; compatible with Hive Metastore API and Iceberg REST.
Where it shines: Enterprises seeking broad governance (tables, files, functions, ML models) and consistent policies across engines/clouds. Notable: Recent updates added external engine read GA and write preview for Iceberg via REST, expanding interoperability.

Lakekeeper

What it is: A lightweight, Rust-based Apache Iceberg REST catalog focused on speed, security (OIDC/OPA), and simplicity; Apache-licensed. Where it shines: Teams wanting a small, fast Iceberg catalog they can self-host, integrate with Trino/Spark, and plug into modern authz. Notable: Ecosystem-first design; good fit for DIY open lakehouses and CICD-style deployments.

Quick Guide: Picking the Right Catalog

Open, vendor-neutral Iceberg core: Polaris (add Nessie if you need Git-style branching & multi-table commits).
Federate many sources/catalogs across regions/clouds: Gravitino.
Deep cloud integration: Glue/Lake Formation (AWS), OneLake Catalog (Azure/Fabric), BigLake Metastore (GCP) or Dremio Catalog (Managed Polaris Service from Dremio).
Broad data & AI governance (tables + files + models): Unity Catalog (OSS); growing multi-engine support including Iceberg REST.

Tip: For multi-engine Iceberg lakehouses, a common pattern is: Polaris as the primary REST catalog for engines, with Nessie layered in when you need branches/isolated environments. Cloud-native teams may still register those tables in their cloud catalogs for service-level features (e.g., Athena/BigQuery/Power BI), but keep the source of truth open.

Lakehouse Optimization

Once a lakehouse is in production, the focus shifts from building to sustaining performance and efficiency at scale. Without ongoing optimization, query times creep up, storage costs balloon, and data reliability weakens. The key is to manage both physical data layout and metadata growth.

Compaction and Small File Management

Frequent batch loads and streaming pipelines often generate thousands of small Parquet or ORC files.

Problem: Query engines spend more time opening files than scanning data.
Solution: Table formats support compaction actions, rewriting many small files into fewer large ones (hundreds of MBs to 1GB).
Examples:
- Iceberg’s rewriteDataFiles action merges small files efficiently (Dremio also has an OPTIMIZE command for Iceberg tables).
- Delta Lake offers the OPTIMIZE command (with Z-Ordering for clustering).
- Hudi provides asynchronous background compaction for MoR tables.

Snapshot Expiration and Metadata Cleanup

Modern formats keep snapshots for time travel, but unchecked, these create metadata bloat.

Iceberg: expireSnapshots safely removes old snapshots and associated data files.
Delta Lake: VACUUM cleans up unreferenced files after a retention period.
Hudi: Timeline service supports configurable retention of commits and delta logs.
Regular cleanup keeps both storage and query planning efficient.

Partitioning and Clustering

Good partition design reduces data scanned per query.

Iceberg: Hidden partitions abstract complexity away from end users.
Delta: Z-Ordering clusters data across dimensions for multidimensional pruning.
Hudi: Can cluster records within files to optimize MoR query performance.

Partition evolution, changing partition strategy over time without breaking old data, is now supported by most formats and prevents schema rigidity.

Query Acceleration

Beyond storage optimization, query acceleration techniques deliver speed at scale.

Dremio's Reflections and materialized views in platforms like Dremio provide always-fresh, cache-like performance boosts without manual tuning.
Column stats and bloom filters stored in metadata allow engines to skip files entirely when filters exclude them.
Vectorized execution and Arrow-based memory models reduce CPU costs across query engines.

Format-Specific Optimizations

Iceberg: Manifest merging, hidden partitioning, and metadata caching.
Delta: Frequent checkpoints and file skipping using data skipping indexes.
Hudi: Incremental queries for consuming only new changes.
Paimon: Continuous compaction to reconcile streaming write amplification.

Bottom line: Lakehouse optimization is not a one-off task, it’s an ongoing discipline. By managing file sizes, pruning metadata, evolving partitions, and using acceleration features, organizations keep performance predictable and costs controlled, even as data volumes and workloads scale into 2026.

The Intelligent Data Lakehouse Built for Agentic AI with Dremio

By the end of 2025, the conversation about data platforms shifted from “how do we manage data?” to “how do we make data intelligent and AI-ready?” This is where the intelligent lakehouse comes in, and where Dremio stands out as the reference implementation.

From Static Analytics to Agentic Workloads

Traditional BI queries are predictable: weekly reports, dashboards, and KPIs. AI-driven workloads are not. Agentic AI systems, large language models and autonomous agents, generate dynamic, ad-hoc queries that span datasets in unpredictable ways. This requires:

Consistent low-latency responses.
A platform that can optimize itself without human intervention.
Seamless integration between structured data, semantic meaning, and AI agents.

Dremio as the Intelligent Lakehouse

Dremio is more than a query engine; it’s a self-optimizing lakehouse platform built natively on Apache Iceberg and Arrow. Key capabilities include:

Reflections: Always-fresh materializations that accelerate queries automatically. Unlike traditional materialized views, reflections are invisible to end users, the optimizer decides when to use them, making acceleration adaptive to changing workloads.
Semantic Layer: A unified place to define datasets, metrics, and business concepts. This ensures that whether it’s an analyst writing SQL or an AI agent generating queries, results remain consistent and governed.
AI-Ready APIs: Through Arrow Flight and REST endpoints, Dremio streams data directly into Python, notebooks, or AI frameworks with zero-copy efficiency. This bridges the gap between analytics and machine learning pipelines.
Open Standards: By embracing Iceberg, Polaris (for catalogs), and Arrow, Dremio ensures interoperability, your AI agents or external engines can interact with the same governed data without lock-in.
All the above allow Agentic AI applications connecting to Dremio through Dremio's MCP server successful in enabling Agentic Analytics.

Why This Matters for Agentic AI

AI agents thrive on autonomy and adaptability. They need a platform that:

Handles ever-changing queries without brittle pre-optimizations.
Keeps acceleration aligned with shifting patterns (autonomous reflections).
Provides governed access so that AI doesn’t hallucinate unauthorized or inconsistent definitions of metrics.
Scales seamlessly from small exploratory prompts to massive training-data extractions.

In short: Dremio delivers the intelligent lakehouse, a platform that not only stores and serves data but actively adapts to how humans and AI consume it. As agentic AI moves from hype to everyday practice in 2026, this intelligence layer will be the key to transforming raw data into reliable, actionable, and AI-ready insights.

Python for the Lakehouse

Python has become the lingua franca of modern data engineering and data science, and the lakehouse ecosystem is no exception. By 2026, a rich set of Python-first tools and frameworks have emerged that make it easier to ingest, process, analyze, and serve data directly from open table formats like Apache Iceberg, Delta, and Hudi. These tools not only enable lightweight experimentation but also power production-grade pipelines that rival traditional big data stacks.

DuckDB

Often described as the “SQLite for analytics,” DuckDB is an in-process analytical database that excels at local workloads:

Direct Parquet & Iceberg Reads: DuckDB can query Parquet files and integrate with Iceberg catalogs, making it a natural fit for small-to-medium lakehouse use cases.
Speed: Its vectorized execution engine makes it extremely fast for analytical queries on a single machine.
Python Integration: Native bindings allow seamless use within notebooks or Python apps.

DuckDB has become the go-to for prototyping, ad hoc exploration, and embedding analytics directly into applications.

Dask

Dask is a parallel computing framework for Python that scales workflows from laptops to clusters.

Flexible API: Works with familiar NumPy, pandas, and scikit-learn APIs while distributing workloads.
Lakehouse Integration: Reads and writes Parquet, and combined with Iceberg connectors, it enables scalable transformations on lakehouse data.
Ecosystem Fit: Useful for machine learning preprocessing and large-scale data transformations where Spark might be overkill.

Dask democratizes distributed compute for teams already invested in Python.

Daft

A newer entrant, Daft positions itself as a distributed data processing engine optimized for AI and ML workloads.

Arrow-Native: Built on Apache Arrow for fast columnar in-memory processing.
Flexible Backends: Runs locally or on clusters, supporting both CPUs and GPUs.
Lakehouse Ready: Reads directly from Parquet and Iceberg sources, enabling high-performance pipelines that integrate analytics and ML training.

Daft is gaining traction for teams that want a modern, Pythonic alternative to Spark for big data and AI-centric workflows.

Bauplan

Bauplan Labs brings a serverless, Python-first lakehouse approach.

Pipeline-as-Code: Data pipelines are written in Python and executed in a serverless runtime that scales automatically.
Version Control for Data: Bauplan integrates Iceberg tables with Git-like branching via catalogs like Nessie, making schema and data versioning first-class features.
Developer Experience: With Arrow under the hood, Bauplan emphasizes reproducibility, modular pipelines, and minimal infrastructure overhead.

Bauplan is designed for teams that want the power of the lakehouse without the complexity of managing heavy infrastructure.

In practice:

DuckDB is the Swiss Army knife for local analytics.
Dask scales familiar Python workflows across clusters.
Daft brings Arrow-native distributed compute optimized for AI.
Bauplan simplifies pipeline execution with a serverless lakehouse model.

Together, these tools give Python along other libraries like Polaris, Ibis, SQLFrame and others developers an end-to-end toolkit for building, maintaining, and consuming modern data lakehouses.

Graphs in the Data Lakehouse with PuppyGraph

While the data lakehouse excels at tabular and relational analytics, many real-world problems are graph-shaped: fraud rings, identity networks, supply chains, lineage tracking, and recommendation systems. Traditionally, these problems required loading data into a specialized graph database, an extra layer of ETL and storage that added cost and complexity. PuppyGraph changes this equation by bringing graph analytics directly into the lakehouse.

What is PuppyGraph?

PuppyGraph is a cloud-native graph engine designed to run on top of existing data in your lakehouse. Instead of requiring a proprietary graph database, PuppyGraph lets you query your Iceberg, Delta, Hudi, or Hive tables as a graph. It connects directly to open table formats, relational databases, and warehouses, automatically sharding and scaling queries without duplicating data. This means you can turn your existing datasets into a graph model in minutes, with no ETL.

Integration with the Lakehouse

PuppyGraph integrates seamlessly with:

Apache Iceberg (including REST catalogs like Tabular or Polaris)
Delta Lake and Apache Hudi
Hive Metastore and AWS Glue
Databases and Warehouses such as PostgreSQL, MySQL, Redshift, BigQuery, and DuckDB

Each source is treated as a catalog. PuppyGraph lets you define a graph schema across one or many catalogs, effectively federating multiple data sources into a single graph. For example, you can link customer nodes in PostgreSQL with transaction edges in Iceberg, all without moving the data.

Querying Graphs at Scale

Because PuppyGraph queries directly against Parquet-backed tables, you can run multi-hop traversals and graph algorithms over your lakehouse data. It supports popular graph query languages:

Gremlin (Apache TinkerPop)
openCypher

This ensures compatibility with existing graph tooling and reduces the learning curve. Performance is optimized for large, complex traversals: PuppyGraph has demonstrated 6-hop traversals over hundreds of millions of edges in under a second. Cached mode allows even faster repeated queries, often surpassing the performance of traditional graph databases.

Use Cases

Fraud Detection: Traverse transaction graphs in real time to uncover hidden fraud rings.
Cybersecurity: Model logins, access patterns, and network flows as a graph to detect threats.
Supply Chain Optimization: Connect suppliers, shipments, and logistics into a graph for bottleneck analysis.
Customer 360: Combine relational and behavioral data into a graph to better understand customer journeys.
Graph + AI: PuppyGraph supports Graph RAG (Retrieval Augmented Generation), enabling LLMs and agents to query structured relationships for better context and reasoning.

Why It Matters

By plugging directly into the lakehouse, PuppyGraph removes the wall between tabular and graph analytics. Data engineers and architects can:

Avoid data duplication and ETL pipelines into separate graph stores.
Keep governance and security consistent via existing catalogs.
Support SQL, BI, and graph queries side-by-side on the same data.

In essence, PuppyGraph makes the lakehouse not just the foundation for relational analytics and AI, but also a native home for graph workloads, all with the same open formats and scalable storage.

Edge Inference for the Lakehouse: Spice AI

As organizations embrace AI-driven applications, the edge has become a critical deployment target. Instead of sending all data to centralized clusters, inference can increasingly happen close to where data is generated, IoT devices, factories, mobile applications, or regional data centers. The lakehouse, traditionally viewed as a central hub, is now extending outward. Platforms like Spice AI make this possible.

Why Edge Inference Matters

Latency: Inference needs to happen in milliseconds, not seconds. Shipping every query to the cloud adds unacceptable delays for use cases like predictive maintenance or fraud detection.
Cost Efficiency: Processing locally reduces bandwidth and cloud compute costs, especially when dealing with high-volume sensor or event data.
Resilience: Edge inference continues to function even with intermittent network connectivity, syncing back to the lakehouse when available.
Privacy & Compliance: Processing data locally helps meet regulatory requirements by minimizing the movement of sensitive information.

Spice AI at the Edge

Spice AI positions itself as an operational data lakehouse tailored for real-time and AI workloads. At the edge, this means:

Federated Querying with DataFusion: Spice uses the Rust-based DataFusion engine (part of the Arrow ecosystem) to execute high-performance queries locally. This allows lightweight nodes to join, filter, and aggregate data directly.
Vector + Relational Search: Spice combines vector search (for embeddings) with SQL-style queries. This means an edge application can run both semantic AI lookups and structured analytics in one step.
Lightweight Runtimes: Spice can run in containers or edge environments, consuming a small footprint while still supporting open table formats like Iceberg, Delta, and Hudi.
Hybrid Sync: Results and inferences can be materialized locally, then synchronized back to the central lakehouse when connectivity is restored, ensuring global consistency without sacrificing local responsiveness.

Example Use Cases

Manufacturing IoT: Edge devices monitor sensor streams, detect anomalies with on-device inference, and sync flagged events to the lakehouse for broader analysis.
Retail: In-store applications recommend products in real time based on customer behavior while syncing aggregated insights centrally.
Telecom/5G: Local edge inference supports real-time network optimization while global models are trained and governed in the lakehouse.

In summary: Edge inference extends the reach of the lakehouse from the cloud to the edge, enabling AI applications to be both real-time and governed.

DuckLake: Simplifying Lakehouse Metadata with SQL

While formats like Iceberg, Delta, and Hudi advanced the lakehouse by bringing ACID transactions to data lakes, they also introduced operational complexity: JSON manifests, Avro metadata files, separate catalog services, and eventual consistency challenges. DuckLake takes a fresh approach by asking a simple question: what if the entire metadata layer was just stored in a relational database?

What is DuckLake?

DuckLake is a new open table format developed by the DuckDB team. Its core idea is to move all catalog and table metadata into a SQL database, while keeping table data as Parquet files in object storage or local filesystems. This means no manifest lists, no Hive Metastore, and no extra catalog API services—just SQL tables that track schemas, snapshots, and file pointers.

Architecture

DuckLake splits the lakehouse into two layers:

Catalog Database: Any ACID-compliant database (DuckDB, SQLite, Postgres, MySQL, or even MotherDuck) stores all metadata—schemas, table versions, statistics, and transactions.
Data Layer: Standard Parquet files (and optional delete files) stored in directories or S3 buckets hold the actual table data.

This design yields fast commits (a single SQL transaction to update metadata), strong consistency (no reliance on eventually consistent file stores), and simpler operations (just back up or replicate the metadata DB). It also enables advanced features like multi-table transactions, time travel, and transactional schema changes without a complex stack.

Integration with DuckDB

DuckLake ships as a DuckDB extension. Once installed, users can:

INSTALL ducklake;
LOAD ducklake;
ATTACH 'ducklake:mycatalog.ducklake' AS lakehouse;

From there, you can create tables, insert, update, delete, and query with full ACID guarantees. Multiple DuckDB instances can share the same DuckLake if the catalog is in a multi-user database like Postgres, effectively making DuckDB “multiplayer” with a shared lakehouse.

Interoperability

DuckLake is its own format, but it’s designed to interoperate:

Iceberg: Parquet and delete files are compatible, and DuckLake can import Iceberg metadata directly, even preserving snapshot history.
Delta: DuckDB continues to support Delta Lake separately; data can be copied between Delta and DuckLake when needed.
Hudi: Not natively supported yet, but Hudi’s Parquet files can be queried as plain Parquet.

This makes DuckLake a flexible companion in mixed-format environments and a potential bridge for migrating or experimenting.

Use Cases

Local & Embedded Lakehouses: Run a mini data warehouse on your laptop with DuckDB + DuckLake, no heavy services required.

Small Team Data Warehouses: Share a DuckLake catalog in Postgres for concurrent analytics across a team.
Streaming & CDC: Handle high-frequency small writes efficiently without metadata file bloat.
CI/CD Pipelines: Spin up ephemeral lakehouses in tests with time travel and rollback for validation.

Limitations & Roadmap

DuckLake is still young (v0.3 as of late 2025). At present:

Ecosystem support is centered on DuckDB/MotherDuck.
No built-in fine-grained governance; relies on the underlying DB’s permissions.
No branching/merge semantics like Project Nessie, though time travel is supported..

Books on the Data Lakehouse and Open Table Formats

For data engineers and architects who want to go beyond blogs and documentation, books provide the depth and structured learning needed to master the lakehouse paradigm. Between 2023 and early 2026, O’Reilly, Manning, and Packt have released (or announced) a range of titles that cover the architecture, theory, and practice of the data lakehouse, including the major open table formats, Apache Iceberg, Delta Lake, and Apache Hudi.

O’Reilly Media

Apache Iceberg: The Definitive Guide – Data Lakehouse Functionality, Performance, and Scalability on the Data Lake
Tomer Shiran, Jason Hughes, and Alex Merced (Jun 2024)
Comprehensive deep dive into Apache Iceberg’s architecture, metadata model, features like partition evolution and time travel, and integrations across engines such as Spark, Flink, Trino, and Dremio.
Apache Polaris: The Definitive Guide – Enriching Apache Iceberg Lakehouse with a robust open-source catalog
Alex Merced, and Andrew Madson (Jun 2024)
Revolutionize your understanding of modern data management with Apache Polaris (incubating), the open source catalog designed for data lakehouse industry standard Apache Iceberg. This comprehensive guide takes you on a journey through the intricacies of Apache Iceberg data lakehouses, highlighting the pivotal role of Iceberg catalogs.
Delta Lake: Up and Running – Modern Data Lakehouse Architectures with Delta Lake
Bennie Haelen and Dan Davis (Oct 2023)
Introductory and practical guide to Delta Lake, covering ACID transactions, schema enforcement, time travel, and how to build reliable data pipelines that unify batch and streaming.
Delta Lake: The Definitive Guide – Modern Data Lakehouse Architectures with Data Lakes
Denny Lee, Tristen Wentling, Scott Haines, and Prashanth Babu (Dec 2024)
Written by core Delta Lake contributors, this book explores Delta’s transaction log, medallion architecture, deletion vectors, and advanced optimization strategies for enterprise-scale workloads.
Practical Lakehouse Architecture: Designing and Implementing Modern Data Platforms at Scale
Gaurav Ashok Thalpati (Aug 2024)
A broad architectural guide to designing, implementing, and migrating to lakehouse platforms. Covers design layers, governance, catalogs, and security with a practical step-by-step framework.
Apache Hudi: The Definitive Guide – Building Robust, Open, and High-Performance Lakehouses
Shiyan Xu, Prashant Wason, Sudha Saktheeswaran, and Rebecca Bilbro (Forthcoming Dec 2025)
Focuses on Hudi’s approach to incremental processing, upserts/deletes, clustering, and indexing. Demonstrates how to run production-ready lakehouses with streaming data ingestion.

Manning Publications

Architecting an Apache Iceberg Lakehouse
Alex Merced (MEAP, 2025 – Forthcoming 2026)
A hands-on, architecture-first guide to designing scalable Iceberg-based lakehouses. Covers all five layers (storage, table formats, ingestion, catalog, consumption) with exercises and real-world design trade-offs.

Packt Publishing

Building Modern Data Applications Using Databricks Lakehouse
Will Girten (Oct 2024)
Practical guide to deploying end-to-end pipelines on Databricks Lakehouse using Delta Lake and Unity Catalog, including batch and streaming workflows, governance, and CI/CD.
Engineering Lakehouses with Open Table Formats
Dipankar Mazumdar and Vinoth Govindarajan (Dec 2025)
Covers Iceberg, Hudi, and Delta Lake together, focusing on how to choose between them, optimize tables, and build interoperable, vendor-agnostic architectures. Includes hands-on examples with Spark, Flink, and Trino.
Data Engineering with Databricks Cookbook
Pulkit Chadha (May 2024)
Recipe-based approach to building data pipelines on Databricks, with step-by-step instructions for managing Delta Lake tables, handling streaming ingestion, orchestrating workflows, and applying Unity Catalog governance.

Takeaway

Whether you’re looking for a deep dive into a specific format (Iceberg, Delta, or Hudi) or a broader perspective on lakehouse architecture, these titles form the essential reading list for data engineers and architects in 2026. They not only document the current state of the technology but also provide practical frameworks and best practices to implement reliable, scalable, and open lakehouses.

Conclusion

The journey from data warehouses to data lakes and finally to the data lakehouse reflects one constant: organizations need a platform that balances trust, flexibility, and performance. Warehouses gave us governance but lacked agility. Lakes gave us scale and freedom but sacrificed reliability. The lakehouse unites these worlds by layering open table formats, catalogs, and intelligent query engines on top of low-cost object storage.

By 2025, this model matured from a promise into a proven architecture. With formats like Apache Iceberg, Delta Lake, Hudi, and Paimon, data teams now have open standards for transactional data at scale. Streaming-first ingestion, autonomous optimization, and catalog-driven governance have become baseline requirements. Looking ahead to 2026, the lakehouse is no longer just a central repository, it extends outward to power real-time analytics, agentic AI, and even edge inference.

For data engineers and architects, the message is clear:

Adopt open table formats to avoid lock-in and ensure interoperability.
Embrace a layered architecture that separates storage, metadata, ingestion, catalog, and consumption.
Optimize continuously, through compaction, snapshot expiration, and acceleration features, so performance scales with data.
Prepare for the future where AI workloads are not occasional but constant, demanding a platform that is both intelligent and adaptive.

The lakehouse has become the backbone of modern data platforms. As you step into 2026, building on this foundation isn’t just a best practice, it’s the path to delivering data that is truly trusted, governed, and AI-ready.

Composable Analytics with Agents - Leveraging Virtual Datasets and the Semantic Layer

Alex Merced — Wed, 17 Sep 2025 09:00:00 GMT

The promise of AI in analytics isn’t just faster answers, it’s smarter, more flexible insights. For that to happen, AI agents need not only access to data but also the ability to compose, extend, and recombine datasets on the fly. This is where Dremio’s semantic layer and virtual datasets come into play, providing the foundation for what AtScale calls composable analytics.

The Challenge: Static Models in a Dynamic World

Traditional analytics models are rigid. Business intelligence teams define metrics in dashboards or cubes, and changing them often requires IT involvement. This creates bottlenecks when business needs evolve, leaving AI agents with limited flexibility to adjust their workflows.

For agentic AI, which thrives on iterative reasoning and adaptive workflows, rigid models are a barrier.

Virtual Datasets: Building Blocks for Composable Analytics

Dremio addresses this challenge with virtual datasets (VDSs):

No physical copies: VDSs are views defined in the semantic layer, not duplicated data.
Composable: VDSs can be combined, extended, or refined into new virtual models.
Governed: Every dataset inherits security and lineage from the semantic layer.

Agents interacting through Dremio’s MCP server can query these VDSs directly, creating new analytic combinations without breaking governance or requiring new pipelines.

Agents + MCP: Extending Models on Demand

With MCP exposing tools like Run SQL Query and Run Semantic Search, agents can:

Discover governed VDSs in plain business language.
Combine datasets to answer multi-dimensional questions.
Extend existing models with new calculations or filters.

For example, an agent could take a “Customer Revenue” VDS and extend it with a churn prediction metric, producing a new analytic model for marketing, all governed by Dremio’s semantic layer.

Composable Analytics Meets Composable Modeling

The AtScale community describes composable analytics as the ability to assemble insights from modular building blocks. Dremio’s semantic layer aligns perfectly with this vision:

Reusability: Metrics and datasets defined once can be reused everywhere.
Cross-functional consistency: Finance, marketing, and operations share the same definitions.
Agent empowerment: AI systems don’t just query data — they can compose new insights dynamically.

This brings composability from the human analyst’s world into the AI agent’s world.

Real-World Benefits

Faster iteration: Agents adapt models to new questions without waiting for IT.
Democratized insights: Business teams get answers in language they understand, grounded in governed metrics.
Cross-functional alignment: Everyone — human or agent — works from the same semantic foundation.

The result is analytics that are not only AI-ready but also flexible, governed, and consistent across the enterprise.

Conclusion

Composable analytics is the future of data-driven decision-making. By leveraging virtual datasets and the semantic layer, Dremio makes it possible for both humans and AI agents to build and extend insights in real time.

With MCP providing the bridge and the semantic layer ensuring governance, enterprises can embrace a world where analytics are adaptive, modular, and truly agentic.

The Endgame — Building an Autonomous Optimization Pipeline for Apache Iceberg

Alex Merced — Tue, 16 Sep 2025 09:00:00 GMT

The Endgame — Building an Autonomous Optimization Pipeline for Apache Iceberg

Over the past nine posts, we’ve walked through the strategies, techniques, and tools you can use to keep your Apache Iceberg tables optimized for performance, cost, and reliability. Now, it’s time to put it all together.

In this final post of the series, we’ll explore how to build an autonomous optimization pipeline—a system that intelligently monitors your Iceberg tables and triggers the right actions automatically, without manual intervention.

What Does Autonomous Optimization Look Like?

An autonomous pipeline for Iceberg optimization should:

Continuously monitor table metadata
Detect symptoms of degradation (e.g., small files, bloated manifests)
Dynamically trigger the right optimization actions
Recover gracefully from failure
Integrate seamlessly with ingestion and query operations

This makes your lakehouse self-healing, scalable, and easier to maintain—especially across many datasets.

Core Components of the Pipeline

1. Metadata Intelligence Layer

Leverage Iceberg’s built-in metadata tables to:

Analyze file sizes and counts
Track snapshot growth
Monitor partition health
Flag layout drift (e.g., outdated sort orders or clustering)

Example diagnostic query:

SELECT partition, COUNT(*) AS file_count, AVG(file_size_in_bytes) AS avg_file_size
FROM my_table.files
GROUP BY partition
HAVING COUNT(*) > 20 AND AVG(file_size_in_bytes) < 128000000;

This layer becomes the decision-maker for whether compaction or cleanup is needed.

2. Orchestration Layer

Use a scheduling tool like Airflow, Dagster, or dbt Cloud to:

Run diagnostic checks on a schedule
Execute Spark/Flink optimization jobs conditionally
Log and track outcomes
Handle retries and alerting
A sample DAG might include:
check_small_files task
trigger_compaction task
expire_snapshots task
rewrite_manifests task

Each can be run only if certain thresholds are met.

3. Execution Layer

Trigger physical optimizations using:

Spark actions (RewriteDataFiles, ExpireSnapshots, RewriteManifests)
Flink background jobs (especially for streaming pipelines)
Dremio OPTIMIZE and VACUUM

All actions should be:

Scoped to affected partitions
Tuned for parallelism
Capable of partial progress

4. Observability and Logging

Feed metrics into dashboards and alerts using tools like:

Prometheus + Grafana
Datadog
CloudWatch

Track:

Number of files compacted
Snapshots expired
Runtime per job
Failed vs succeeded partitions

This allows you to adjust thresholds and tuning parameters over time.

5. Storage Cleanup (GC)

After snapshots are expired, unreferenced files need to be deleted.
Ensure cleanup happens after expiration jobs, not in parallel.

Benefits of an Autonomous Pipeline

Consistent Performance: Tables stay fast without manual tuning

Operational Efficiency: No more ad hoc optimization jobs

Scalability: Works across 10 tables or 10,000 tables

Governance-Ready: All changes are tracked, repeatable, and policy-driven

Final Thoughts

Iceberg's flexibility and rich metadata layer make it uniquely suited to autonomous data management. By combining:

Real-time metadata insight
Targeted optimization strategies
Smart orchestration
Catalog-aware execution

You can build a lakehouse that optimizes itself—freeing your data team to focus on innovation, not maintenance.

Where to Go from Here

If you’ve followed this series from the beginning, you now have:

A deep understanding of how Iceberg tables degrade
Tools to address compaction, clustering, and metadata bloat
The blueprint for a modern, self-tuning optimization pipeline

Thanks for reading—and keep building faster, cleaner, and smarter Iceberg lakehouses.

Managing Large-Scale Optimizations — Parallelism, Checkpointing, and Fail Recovery

Alex Merced — Tue, 09 Sep 2025 09:00:00 GMT

Managing Large-Scale Optimizations — Parallelism, Checkpointing, and Fail Recovery

When working with Apache Iceberg at scale, optimization jobs can become heavy and time-consuming. Rewriting thousands of files, scanning massive partitions, and coordinating metadata updates requires careful execution planning—especially in environments with limited compute or strict SLAs.

In this post, we’ll look at strategies for making compaction and metadata cleanup operations scalable, resilient, and efficient, including:

Tuning parallelism
Using partition pruning
Applying checkpointing for long-running jobs
Handling failures safely and automatically

Why Scaling Optimization Matters

As your Iceberg tables grow:

File counts increase
Partition cardinality rises
Manifest files balloon
Compaction jobs touch terabytes of data

Without scaling strategies:

Jobs may fail due to timeouts or memory errors
Optimization may lag behind ingestion
Query performance continues to degrade despite efforts

1. Leveraging Partition Pruning

Partition pruning ensures that only the parts of the table that need compaction are touched.

Use metadata tables to target only problem areas:

SELECT partition
FROM my_table.files
GROUP BY partition
HAVING COUNT(*) > 20 AND AVG(file_size_in_bytes) < 100000000;

You can then pass this list to a compaction job to limit the scope of the rewrite.

2. Tuning Parallelism in Spark or Flink

Large optimization jobs should run with enough parallel tasks to distribute I/O and computation.

In Spark: Use spark.sql.shuffle.partitions to increase default parallelism.

Tune executor memory and cores to handle larger partitions.

Use .option("partial-progress.enabled", true) for better resilience in Iceberg actions.

spark.conf.set("spark.sql.shuffle.partitions", "200")

Actions.forTable(spark, table)
  .rewriteDataFiles()
  .option("min-input-files", "5")
  .option("partial-progress.enabled", "true")
  .execute()

In Flink:

Use fine-grained task managers
Enable incremental compaction and checkpointing

3. Incremental and Windowed Compaction

Don’t try to compact the entire table at once. Instead:

Group partitions into batches
Use rolling windows (e.g., compact N partitions per hour)
Resume from the last successfully compacted partition on failure
You can build this logic into orchestration tools like Airflow or Dagster.

4. Checkpointing and Partial Progress

Iceberg supports partial progress mode in Spark:

.option("partial-progress.enabled", "true")

This allows successfully compacted partitions to commit, even if others fail—making retries cheaper and safer.

In Flink, this is handled more granularly via stateful streaming checkpointing.

5. Retry and Failover Strategies

Wrap compaction logic in robust retry mechanisms:

Use exponential backoff
Separate retries by partition
Alert on repeated failures for human intervention

For example, in Airflow:

PythonOperator(
    task_id="compact_partition",
    python_callable=run_compaction,
    retries=3,
    retry_delay=timedelta(minutes=5)
)

Also consider:

Writing logs to object storage for audit
Emitting metrics to Prometheus/Grafana for observability

6. Monitoring Job Health

Track:

Job duration
Files rewritten vs skipped
Failed partitions
Number of manifests reduced
Snapshot size pre- and post-job
These metrics help tune parameters and detect regressions over time.

Summary

Scaling Iceberg optimization jobs requires thoughtful execution planning:

Use metadata to limit scope
Tune parallelism to avoid resource waste
Use partial progress and checkpointing to survive failure
Automate retries and monitor outcomes

In the final post of this series, we’ll bring it all together—showing how to build a fully autonomous optimization pipeline using orchestration, metadata triggers, and smart defaults.

Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

Alex Merced — Fri, 05 Sep 2025 09:00:00 GMT

Free Resources

Agentic AI is quickly moving from the whiteboard to production. These aren’t just smarter chatbots—they're intelligent systems that reason, learn, and act with autonomy. They summarize research, manage operations, and even coordinate complex workflows. But while models have become more capable, they still hit a wall without the right data infrastructure.

That wall? It's not just about storage—it's about access, performance, and context.

Many organizations building AI agents find themselves struggling with data silos, unpredictable performance, and a lack of clarity around what the data actually means. The result? Agents that stall, generate shallow results, or make the wrong decisions altogether.

To unlock the full potential of Agentic AI, we need to rethink how our data platforms are designed. This is where Apache Iceberg and Dremio come in. Together, they provide a modern, open lakehouse architecture that solves the three core bottlenecks to AI success:

Frictionless access to enterprise data (without data wrangling or replication)
Autonomous, high-performance query acceleration (built for dynamic workloads)
A semantic layer that gives agents the context they need to understand and act

In this post, we’ll break down each of these challenges—and show how Iceberg and Dremio together build the intelligent data backbone your AI agents need to thrive.

The 3 Bottlenecks Blocking Agentic AI from Delivering Real Impact

As promising as Agentic AI is, most organizations hit the same three roadblocks on the path to real-world success. These aren't just technical hurdles—they're architectural challenges that undermine the speed, accuracy, and reliability of intelligent agents.

Let’s break them down:

1. Access to Data: Silos, Bottlenecks, and Delays

AI agents need a holistic view of your enterprise to operate effectively—marketing data, operational logs, customer records, product telemetry, and more. But in most environments, that data is scattered across:

Cloud storage systems
Operational databases
SaaS platforms
Departmental data warehouses

Each of these systems may have different governance rules, inconsistent formats, or delayed ETL pipelines. Worse, getting access often requires waiting on central data teams or replicating data manually. This slows down experimentation and limits what your agents can “see.”

2. Performant Access: When Every Millisecond Counts

Even when agents can access data, they still need it fast. AI workflows—especially agentic ones—are dynamic and unpredictable. One minute it’s a lookup query; the next it’s a multi-join aggregation across several sources. Traditional performance tuning—manual partitioning, index maintenance, and query tuning—can’t keep up.

Agents can’t wait minutes for answers. They need sub-second response times to chain actions together effectively. Without autonomous performance management, latency becomes a dealbreaker.

3. Semantic Meaning: Knowing What the Data Actually Means

Access and speed are critical—but so is understanding. AI agents need context to interpret data correctly. What does customer_type = 2 actually mean? Is “margin” defined the same way in marketing and finance? Without a shared semantic layer, agents operate on guesswork.

This is where many AI initiatives fail quietly. Outputs look correct on the surface but are misaligned with how the business actually thinks about its data.

Solving these challenges requires more than patchwork fixes. It demands a new kind of data architecture—one that is open, intelligent, and built for automation. And that’s where Apache Iceberg and Dremio make all the difference.

Apache Iceberg: The Open Foundation for AI-Ready Data

When it comes to building a scalable, AI-optimized data platform, Apache Iceberg is the backbone that holds it all together. It’s not just another table format—it’s the evolution of how data is organized, versioned, and accessed in modern analytics and AI environments.

Think of Iceberg like the index in a giant filing cabinet. It doesn’t just store your data—it brings order, consistency, and flexibility to your data lake, making it feel like a fully featured data warehouse without giving up the openness of object storage.

Why Apache Iceberg Matters for Agentic AI

Agentic AI requires access to data that is:

Consistent: So the same query always returns the same answer.
Evolvable: So schema changes don’t break downstream pipelines.
Portable: So any tool—Spark, Flink, Dremio, or even your AI agents—can access it without vendor lock-in.

Apache Iceberg delivers all of this with features like:

Schema evolution: Add, drop, rename columns without rewriting data.
Time travel: Query data “as of” any point in time, ideal for audits or AI state comparisons.
Hidden partitioning: Optimize performance without complicating your SQL.
ACID transactions: Ensure atomic, consistent updates in multi-writer environments.

By standardizing on Iceberg, your organization can avoid the “tool wars” between departments. Everyone works from the same data foundation, using the tools they prefer—whether it’s SQL notebooks, BI dashboards, or LLM-powered agents.

The Lakehouse Advantage

Iceberg unlocks the full potential of the lakehouse model: combining the flexibility of a data lake with the performance and structure of a data warehouse. This modular approach means:

Teams aren’t forced to centralize around one compute engine.
You avoid redundant data copies and ETL pipelines.
AI agents can query directly from the lakehouse with open standards.

In short, Apache Iceberg makes your data open, unified, and production-grade—everything your intelligent agents need to act with confidence.

Dremio: The Intelligent Data Interface for Agentic AI

Apache Iceberg gives you the open foundation—but Dremio turns that foundation into an intelligent, AI-ready platform. Think of Dremio as the control plane that gives both humans and AI agents seamless access to the data they need, with speed, security, and semantic understanding built in.

Let’s explore how Dremio removes the remaining friction across access, performance, and context.

Unified Access Across All Data (Federation + Simplified Governance)

Even in the best-case scenario, not all your data will live in Iceberg tables. You still have data in relational databases, SaaS tools, cloud data warehouses, and more.

This is where Dremio’s Zero-ETL Federation shines. Dremio connects directly to all your sources—whether it’s Amazon S3, PostgreSQL, Salesforce, or MongoDB—and lets you query them in place, without copying data or building fragile pipelines.

Benefits for Agentic AI:

Agents can query the full landscape of enterprise data through a single interface.
Centralized access control means fewer credentials to manage or expose.
Real-time insights from operational systems without waiting on ingestion jobs.

Autonomous Performance for Unpredictable Workloads

Agentic AI is dynamic by nature—queries change based on real-time decisions. You can't rely on hand-tuned optimizations or static dashboards.

Dremio solves this with autonomous performance management, including:

Automatic Iceberg Table Optimization: Dremio continuously compacts small files, sorts data, and maintains metadata health to reduce I/O and boost query speed.
Reflections: Dremio’s version of intelligent materialized views. They’re automatically created, updated incrementally, and substituted at query time—so your agents get faster results without changing their SQL.
Multi-layered Caching: From query plans to result sets to object storage blocks, Dremio caches intelligently to accelerate repeat workloads and reduce cloud costs.

This means whether your AI agent is summarizing a dashboard or crunching through user logs, it gets fast, consistent results—without human intervention.

Built-in Semantic Layer for Shared Understanding

To generate meaningful insights, agents need to understand not just what data is, but what it means. Dremio provides a native semantic layer that bridges that gap.

Semantic Search: Agents and users can discover datasets using natural language.
Data Modeling: Define reusable business logic, KPIs, and metrics as views.
Auto-Generated Wikis: Every dataset can include human-readable descriptions—great for onboarding both analysts and AI systems.
Fine-Grained Access Control: Row- and column-level security ensures agents see only what they’re authorized to.

And with Dremio’s MCP server, your agents can programmatically explore metadata, access semantic context, and generate more accurate queries.

Dremio doesn’t just connect to your data—it understands it, optimizes it, and makes it consumable by anyone (or anything) that needs it. For Agentic AI, this is the difference between guesswork and precision.

Closing the Loop: Iceberg + Dremio = AI-Optimized Lakehouse

When you bring Apache Iceberg and Dremio together, you don’t just get a modern data stack—you get a foundation built for the realities of Agentic AI.

Let’s recap how these technologies align to eliminate the core bottlenecks we explored earlier:

✅ Unlocking Access

Apache Iceberg standardizes how data is stored, making it accessible across tools and teams.
Dremio federates access to all your data sources—cloud, on-prem, SaaS, and more—without the overhead of ETL or manual integration.
AI agents can now query the full enterprise landscape through a single interface, using a single set of credentials, securely and efficiently.

✅ Delivering Performance Autonomously

Iceberg enables high-performance table management (partitioning, file pruning, metadata tracking).
Dremio automates this further—handling compaction, caching, and query acceleration behind the scenes.
Reflections, smart caching, and autonomous query optimization ensure agents get sub-second responses, no matter how complex or spontaneous the query.

✅ Embedding Context Through Semantics

Iceberg brings structure to your data lake, but Dremio gives it meaning.
Through Dremio’s built-in semantic layer and MCP server, your AI agents can interpret, navigate, and reason about data the way your business does.
Whether it’s knowing what “active customer” means or filtering by business unit, Dremio gives your agents the vocabulary to deliver trusted outcomes.

The result is a truly intelligent lakehouse—open, unified, performant, and semantically rich. One that doesn’t just serve humans, but empowers agents to act, adapt, and deliver real business value.

If Agentic AI is your destination, Apache Iceberg and Dremio are the road and the vehicle that will take you there.

Get hands-on with Dremio and Apache Iceberg today and start building the intelligent data foundation your AI agents need to thrive.

Hidden Pitfalls — Compaction and Partition Evolution in Apache Iceberg

Alex Merced — Tue, 02 Sep 2025 09:00:00 GMT

Hidden Pitfalls — Compaction and Partition Evolution in Apache Iceberg

Apache Iceberg offers partition evolution, allowing you to change how your data is partitioned over time without rewriting historical files. This is a major advantage over legacy file formats, but it also introduces new challenges—especially when it comes to compaction and query optimization.

In this post, we’ll explore how partition evolution can impact compaction, metadata management, and query performance—and how to avoid the most common pitfalls.

What Is Partition Evolution?

Partition evolution allows you to:

Add new partition fields
Drop old partition fields
Change partition transforms (e.g., from day(ts) to hour(ts))

Unlike traditional systems that enforce a single static layout, Iceberg lets you evolve the partitioning strategy without rewriting or invalidating historical data.

Example:

-- Original partitioning
ALTER TABLE sales ADD PARTITION FIELD day(order_date);

-- Later evolve to hourly
ALTER TABLE sales DROP PARTITION FIELD day(order_date);
ALTER TABLE sales ADD PARTITION FIELD hour(order_date);

Each snapshot will respect the partition spec that was active at the time the data was written.

The Pitfall: Compaction Across Partition Specs

When compaction jobs span files written under different partition specs, several challenges arise:

File Layout Inconsistency Compaction may combine files that don’t share a common layout, resulting in mixed partition values that reduce query pruning efficiency.
Reduced Predicate Pushdown Query engines rely on partition columns for efficient pruning. If files are mixed across specs, pruning may be incomplete, increasing scan cost.
Compaction Failures or Misbehavior Some engines may fail to rewrite or rewrite files improperly when specs conflict, especially in older versions of Iceberg libraries or poorly configured environments.

Best Practices to Manage Partition Evolution Safely

1. Compact Within Partition Spec Versions

Query the files metadata table to identify which files belong to which spec:

Copy
Edit
SELECT spec_id, COUNT(*) AS file_count
FROM my_table.files
GROUP BY spec_id;

Run compaction per spec_id to preserve consistency and avoid mixing files.

2. Track and Align Sorting and Clustering

When evolving partitions, ensure that sort orders are also updated. Mismatched sort and partition strategies can undermine clustering efforts.

SELECT spec_id, sort_order_id, COUNT(*) 
FROM my_table.files 
GROUP BY spec_id, sort_order_id;

3. Repartition Carefully and Gradually

Avoid abrupt changes like:

Switching from coarse to fine partitioning (e.g., day to minute)
Dropping too many partition fields at once
These can lead to over-fragmentation and more small files unless paired with compaction and sort order realignment.

4. Use Metadata Tables to Guide Evolution

Before evolving a partition spec:

Inspect query patterns (e.g., WHERE clauses)
Evaluate partition sizes and access frequencies
Use tools like Dremio’s catalog lineage and query analyzer if available

5. Communicate Changes Across Teams

If your tables are used across multiple teams or tools:

Document changes to partitioning logic
Include schema and partition spec history in data documentation
Coordinate compaction jobs after major partition changes

Summary

Partition evolution is one of Iceberg’s superpowers—but like all powerful features, it must be used wisely. To avoid performance and optimization issues:

Don’t mix files with different partition specs in compaction jobs
Update sort orders and clustering with partition changes
Monitor partition usage and access patterns continuously

In the next post, we’ll move from structural design to execution tuning—exploring how to scale compaction operations efficiently using parallelism, checkpointing, and fault tolerance.

Using Iceberg Metadata Tables to Determine When Compaction Is Needed

Alex Merced — Tue, 26 Aug 2025 09:00:00 GMT

Using Iceberg Metadata Tables to Determine When Compaction Is Needed

Scheduling compaction at fixed intervals is better than not optimizing at all—but it can still lead to unnecessary compute spend or delayed maintenance. A smarter approach is to dynamically trigger compaction based on real-time metadata signals.

Apache Iceberg makes this possible with its powerful system of metadata tables, which expose granular details about files, snapshots, and manifests.

In this post, we'll explore how to query these tables to:

Detect small files
Identify bloated partitions
Spot manifest inefficiencies
Automate event-driven compaction workflows

What Are Iceberg Metadata Tables?

Every Iceberg table automatically maintains a set of virtual tables that expose its internals. The most relevant for optimization include:

files – List of all data files in the table, including size, partition, and metrics
manifests – List of manifest files and the data files they reference
snapshots – History of table changes and snapshot metadata
history – Timeline of snapshot commits and their lineage

These tables can be queried like any other SQL table, making it easy to introspect your table’s health.

1. Detecting Small Files with the `files` Table

To identify partitions suffering from small file syndrome:

SELECT
  partition,
  COUNT(*) AS file_count,
  AVG(file_size_in_bytes) AS avg_size_bytes
FROM my_table.files
GROUP BY partition
HAVING COUNT(*) > 10 AND AVG(file_size_in_bytes) < 134217728; -- 128 MB

You can use this to:

Trigger compaction on specific partitions
Monitor trends in file size distribution over time

2. Finding Fragmented or Stale Manifests

Bloated metadata can come from too many or inefficient manifest files. Use the manifests table to explore:

SELECT
  COUNT(*) AS manifest_count,
  AVG(added_data_files_count) AS avg_files_per_manifest
FROM my_table.manifests;

Low averages can indicate fragmented manifests that are good candidates for rewriting.

3. Tracking Snapshot Volume and Velocity

To see if snapshots are accumulating too fast (and increasing metadata overhead):

SELECT
  COUNT(*) AS snapshot_count,
  MIN(committed_at) AS first_snapshot,
  MAX(committed_at) AS latest_snapshot
FROM my_table.snapshots;

You can also inspect how many files each snapshot adds or removes to identify noisy patterns from ingestion jobs.

4. Building a Health Score

By combining file count, file size, manifest count, and snapshot frequency, you can compute a "table health score":

-- Example: High file count + small average size = poor health
WITH file_stats AS (
  SELECT COUNT(*) AS total_files, AVG(file_size_in_bytes) AS avg_file_size
  FROM my_table.files
),
manifest_stats AS (
  SELECT COUNT(*) AS total_manifests
  FROM my_table.manifests
)
SELECT
  total_files,
  avg_file_size,
  total_manifests,
  CASE
    WHEN avg_file_size < 67108864 AND total_files > 1000 THEN 'Needs compaction'
    ELSE 'Healthy'
  END AS status
FROM file_stats, manifest_stats;

5. Triggering Compaction Automatically

Once you identify problematic patterns, you can wire up your orchestration layer to act:

Use Airflow, Dagster, or dbt Cloud to run SQL-based checks
When thresholds are breached, trigger Spark/Flink compaction jobs
Track results and update monitoring dashboards
This ensures you optimize only when needed, keeping costs and latency low.

Benefits of Metadata-Driven Optimization

Precision: Only touch affected partitions
Efficiency: Avoid unnecessary compute jobs
Responsiveness: React to real-time ingestion patterns
Governance: Create audit trails for all compaction decisions

Summary

Apache Iceberg gives you visibility and control over your tables through metadata tables. By tapping into this metadata:

You avoid blind scheduling of compaction
You build smarter, more efficient optimization workflows
You reduce both query latency and operational cost

In the next post, we’ll dive into partition evolution and layout pitfalls, and how to avoid undermining your compaction and clustering strategies when schemas or partitions change.

Designing the Ideal Cadence for Compaction and Snapshot Expiration

Alex Merced — Tue, 19 Aug 2025 09:00:00 GMT

Designing the Ideal Cadence for Compaction and Snapshot Expiration

In previous posts, we explored how compaction and snapshot expiration keep Apache Iceberg tables performant and lean. But these actions aren’t one-and-done—they need to be scheduled strategically to balance compute cost, data freshness, and operational safety.

In this post, we’ll look at how to design a cadence for compaction and snapshot expiration based on your workload patterns, data criticality, and infrastructure constraints.

Why Cadence Matters

Without a thoughtful schedule:

Over-optimization can waste compute and create unnecessary load
Under-optimization leads to performance degradation and metadata bloat
Poor coordination can cause clashes with ingestion or query jobs

You need a cadence that fits your data’s lifecycle and your platform’s SLAs.

Key Factors to Consider

1. Ingestion Rate and Pattern

Streaming data? Expect high file churn. Compact frequently (hourly or near-real-time).
Batch jobs? Compact after each large load or on a daily schedule.
Hybrid? Monitor ingestion metrics and trigger compaction based on thresholds.

2. Query Frequency and Latency Expectations

High query volume tables benefit from more frequent compaction to improve scan performance.
Low-usage tables can tolerate more infrequent optimization.

3. Storage Costs and File System Limits

Cloud storage costs can balloon with small files and lingering unreferenced data.
File system metadata limits may also be a concern at massive scale.

4. Retention and Governance Requirements

Snapshots may need to be retained longer for audit or rollback policies.
Balance expiration with compliance needs.

Suggested Cadence Models

Use Case	Compaction Cadence	Snapshot Expiration
High-volume streaming pipeline	Hourly or event-based	Daily, keep 1–3 days
Daily batch ingestion	Post-batch or nightly	Weekly, keep 7–14 days
Low-latency analytics	Hourly	Daily, keep 3–5 days
Regulatory or audited data	Weekly or on-demand	Monthly, retain 30–90 days

Use metadata queries (e.g., from files, manifests, snapshots) to drive dynamic policies.

Automating the Schedule

You can use orchestration tools like:

Airflow / Dagster / Prefect: Schedule and monitor compaction and expiration tasks
dbt Cloud: Use post-run hooks or scheduled jobs to optimize models backed by Iceberg
Flink / Spark Streaming: Trigger compaction inline or via micro-batch jobs

Tip: Tag critical jobs with priorities and isolate them from ingestion workloads where needed.

Coordinating Between Compaction and Expiration

Ideally:

Compact first, then expire snapshots
This ensures snapshots written by compaction are retained at least temporarily
Avoid expiring snapshots too soon after compaction to prevent data loss

Example Workflow:

Run metadata scan to detect small file bloat
Trigger compaction on affected partitions
Delay snapshot expiration by a few hours
Run snapshot expiration with a safety buffer

Monitoring and Adjusting Over Time

Cadence isn’t static—adjust based on:

Changing ingestion rates
New query patterns
Storage trends
Platform feedback (slow queries, GC delays, etc.)

Use logs, metadata tables, and query performance dashboards to guide adjustments.

Summary

An effective compaction and snapshot expiration cadence keeps your Iceberg tables fast, lean, and cost-effective. Your schedule should:

Match your workload patterns
Respect operational and governance needs
Be flexible and monitorable

In the next post, we’ll look at how to use Iceberg’s metadata tables to dynamically determine when optimization is needed—so you can make it event-driven instead of fixed-schedule.

Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests

Alex Merced — Tue, 12 Aug 2025 09:00:00 GMT

Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests

As your Apache Iceberg tables evolve—through continuous writes, schema changes, and compaction jobs—they generate a growing amount of metadata. While metadata is a powerful feature in Iceberg, enabling time travel and auditability, unchecked metadata growth can lead to:

Slower planning and query times
Increased storage costs
Longer table commit and rollback operations
Excessive memory usage during scans

In this post, we’ll explore how to expire old snapshots and rewrite manifests to keep your Iceberg tables lean, responsive, and cost-efficient.

What Causes Metadata Bloat?

Iceberg tracks table state through a series of snapshots. Each snapshot references a set of manifest lists, which in turn reference manifest files describing individual data files.

Bloat occurs when:

Snapshots accumulate and are not expired
Manifests are duplicated across snapshots
Files are replaced by compaction but older snapshots still reference them
Streaming ingestion creates frequent small commits, generating excessive metadata

Expiring Snapshots

You can safely remove older snapshots using Iceberg’s built-in expiration functionality. This deletes metadata for snapshots that are no longer needed for time travel, rollback, or audit purposes.

Example in Spark:

import org.apache.iceberg.actions.Actions

Actions.forTable(spark, table)
  .expireSnapshots()
  .expireOlderThan(System.currentTimeMillis() - TimeUnit.DAYS.toMillis(7)) // keep 7 days
  .retainLast(2) // keep last 2 snapshots no matter what
  .execute();

This keeps recent snapshots while cleaning up older ones, freeing up metadata and unreferenced data files (if garbage collection is also enabled).

Guidelines:

Retain at least a few recent snapshots for rollback safety
Use a time-based and count-based retention policy
Coordinate expiration with your data governance policies

Rewriting Manifests

Over time, manifest files can become inefficient:

Many may reference the same files across snapshots
Some may contain only a few files due to small writes
Their layout may be suboptimal for query planning
You can rewrite manifests to consolidate and reorganize them for improved performance.

Example in Spark:

Actions.forTable(spark, table)
  .rewriteManifests()
  .execute();

This reduces metadata file count, organizes manifests by partition and sort order, and can improve query planning times.

When Should You Perform Metadata Cleanup?

After large ingestion spikes (e.g., backfills)
Following streaming workloads with high commit frequency
Post compaction or schema evolution
On a scheduled basis (e.g., daily or weekly)

Bonus: Use Metadata Tables to Inspect Bloat

Iceberg’s metadata tables help you inspect how much bloat has built up.

Example:

SELECT snapshot_id, added_files_count, total_data_files_count
FROM my_table.snapshots
ORDER BY committed_at DESC;

SELECT COUNT(*) FROM my_table.manifests;

These insights can help you determine when cleanup is needed.

Tradeoffs and Cautions

Snapshot expiration is irreversible: Make sure you don’t need the old snapshots for recovery or audit.
Manifests rewrites are safe but can be compute-intensive on large tables—schedule wisely.
Storage GC may require coordination with your catalog to clean up unreferenced files.

Summary

Metadata is a powerful part of Iceberg’s architecture, but without routine maintenance, it can weigh down your table performance. By:

Expiring stale snapshots
Rewriting bloated manifests
Monitoring metadata tables regularly
You ensure that your Iceberg tables remain agile, scalable, and ready for production workloads.

In the next post, we’ll explore how to design the ideal cadence for compaction and snapshot expiration so your optimizations are timely and cost-effective.

Smarter Data Layout — Sorting and Clustering Iceberg Tables

Alex Merced — Tue, 05 Aug 2025 09:00:00 GMT

Smarter Data Layout — Sorting and Clustering Iceberg Tables

So far in this series, we've focused on optimizing file sizes to reduce metadata and scan overhead. But how data is laid out within those files can be just as important as the size of the files themselves.

In this post, we'll explore clustering techniques in Apache Iceberg, including sort order and Z-ordering, and how these techniques improve query performance by reducing the amount of data that needs to be read.

Why Clustering Matters

Imagine a query that filters on a customer_id. If your data is randomly distributed, every file needs to be scanned. But if the data is sorted or clustered, the engine can skip over entire files or row groups — reducing I/O and speeding up execution.

Clustering benefits:

Fewer files and rows scanned
Better compression ratios
Faster joins and aggregations
More efficient pruning of partitions and row groups

Sorting in Iceberg

Iceberg supports sort order evolution, which lets you define how data should be physically sorted as it's written or rewritten.

You can define sort orders during write or compaction:

import org.apache.iceberg.SortOrder
import static org.apache.iceberg.expressions.Expressions.*;

table.updateSortOrder()
  .sortBy(asc("customer_id"), desc("order_date"))
  .commit();

Use Cases for Sorting

Time-series data: sort by event_time to improve range queries
Dimension filters: sort by commonly filtered columns like region, user_id
Joins: sort by join keys to speed up hash joins and reduce shuffling

Z-order Clustering

Z-ordering is a multi-dimensional clustering technique that co-locates related values across multiple columns. It's ideal for exploratory queries that filter on different combinations of columns.

Example:

table.updateSortOrder()
  .sortBy(zorder("customer_id", "product_id", "region"))
  .commit();

Z-ordering works by interleaving bits from multiple columns to keep related rows close together. This increases the chance that queries filtering on any subset of these columns can benefit from data skipping.

Note: Z-ordering is supported by Iceberg through integrations like Dremio's Iceberg Auto-Clustering and Spark jobs using RewriteDataFiles.

Choosing Between Sort and Z-order

Use Case	Best Technique
Filtering on one key column	Simple Sort
Range queries on timestamps	Sort on time
Multi-column filtering	Z-order
Joins on a key column	Sort on join key
Complex OLAP-style filters	Z-order

When to Apply Clustering

Clustering is typically applied:

During initial writes, if the engine supports it
As part of compaction jobs, using RewriteDataFiles with sort order
In Spark, you can specify sort order in rewrite actions:

Actions.forTable(spark, table)
  .rewriteDataFiles()
  .sortBy("region", "event_time")
  .execute();

Make sure the sort order aligns with your most frequent query patterns.

Tradeoffs

While clustering helps query performance, it comes with tradeoffs:

Sorting increases job duration: Sorting is more expensive than just rewriting files
Clustering can become outdated: Evolving data patterns may require adjusting sort orders
Not all engines respect sort order: Make sure your query engine leverages the layout

Summary

Smart data layout is essential for fast queries in Apache Iceberg. By leveraging sorting and Z-order clustering:

You reduce the volume of data scanned
Improve filter selectivity
Optimize performance for a wide variety of workloads

In the next post, we’ll look at another silent performance killer: metadata bloat, and how to clean it up using snapshot expiration and manifest rewriting.

Optimizing Compaction for Streaming Workloads in Apache Iceberg

Alex Merced — Tue, 29 Jul 2025 09:00:00 GMT

Optimizing Compaction for Streaming Workloads in Apache Iceberg

In traditional batch pipelines, compaction jobs can run in large windows during idle periods. But in streaming workloads, data is written continuously—often in small increments—leading to rapid small file accumulation and tight freshness requirements.

So how do we compact Iceberg tables without interfering with ingestion and latency-sensitive reads? This post explores how to design efficient, incremental compaction jobs that preserve performance without disrupting your streaming pipelines.

The Challenge with Streaming + Compaction

Streaming ingestion into Apache Iceberg often uses micro-batches or event-driven triggers that:

Generate many small files per partition
Write new snapshots frequently
Introduce high metadata churn

A naive compaction job that rewrites entire partitions or the whole table risks:

Commit contention with streaming jobs
Stale data in read replicas or downstream queries
Latency spikes if compaction blocks snapshot availability

The key is to optimize incrementally and intelligently.

Techniques for Streaming-Safe Compaction

1. Compact Only Cold Partitions

Don’t rewrite partitions actively being written to. Instead:

Identify "cold" partitions (e.g., older than 1 hour if partioned by hour)
Compact only those to avoid conflicts with streaming writes

Example query using metadata table:

SELECT partition, COUNT(*) AS file_count
FROM my_table.files
WHERE last_modified < current_timestamp() - INTERVAL '1 hour'
GROUP BY partition
HAVING COUNT(*) > 10;

This can drive dynamic, safe compaction logic in orchestration tools.

2. Use Incremental Compaction Windows

Instead of full rewrites:

Compact only a subset of files at a time (e.g., oldest or smallest)
Avoid reprocessing already optimized files
Reduce job run time to minutes instead of hours

Spark's RewriteDataFiles and Dremio's OPTIMIZE features both support targeted rewrites.

3. Trigger Based on Metadata Metrics

Rather than scheduling compaction at fixed intervals, use metadata-driven triggers like:

Number of files per partition > threshold
Average file size < target
File age > threshold

You can track these via files and manifests metadata tables and use orchestration tools (e.g., Airflow, Dagster, dbt Cloud) to trigger compaction.

Example: Time-Based Compaction Script (Pseudo-code)

# For each partition older than 1 hour with many small files
for partition in get_partitions_older_than(hours=1):
    if count_small_files(partition) > threshold:
        run_compaction(partition)

This pattern allows incremental, scoped jobs that don’t touch fresh data.

Tuning for Performance

Parallelism: Use high parallelism for wide tables to speed up job runtime

Target file size: Stick to 128MB–256MB range unless your queries benefit from larger files

Retries and check-pointing: Make sure jobs are fault-tolerant in production

Summary

To maintain performance in streaming Iceberg pipelines:

Compact frequently, but narrowly
Use metadata to guide scope
Avoid active partitions and large rewrites
Leverage orchestration and branching when available

With the right setup, you can keep query performance and data freshness high—without sacrificing one for the other.

The Basics of Compaction — Bin Packing Your Data for Efficiency

Alex Merced — Tue, 22 Jul 2025 09:00:00 GMT

The Basics of Compaction — Bin Packing Your Data for Efficiency

In the first post of this series, we explored how Apache Iceberg tables degrade when left unoptimized. Now it's time to look at the most foundational optimization technique: compaction.

Compaction is the process of merging small files into larger ones to reduce file system overhead and improve query performance. In Iceberg, this usually takes the form of bin packing — grouping smaller files together so they align with an optimal size target.

Why Bin Packing Matters

Query engines like Dremio, Trino, and Spark operate more efficiently when reading a smaller number of larger files instead of a large number of tiny files. Every file adds cost:

It triggers an I/O request
It needs to be tracked in metadata
It increases planning and scheduling complexity

By merging many small files into fewer large files, compaction directly addresses:

Small file problem
Metadata bloat in manifests
Inefficient scan patterns

How Standard Compaction Works

A typical Iceberg compaction job involves:

Scanning the table to identify small files below a certain threshold.
Reading and coalescing records from multiple small files within a partition.
Writing out new files targeting an optimal size (commonly 128MB–512MB per file).
Creating a new snapshot that references the new files and drops the older ones.

This process can be orchestrated using:

Apache Spark with Iceberg’s RewriteDataFiles action
Dremio with its OPTIMIZE command

Example: Spark Action

import org.apache.iceberg.actions.Actions

Actions.forTable(spark, table)
  .rewriteDataFiles()
  .option("target-file-size-bytes", 134217728) // 128 MB
  .execute()

This will identify and bin-pack small files across partitions, replacing them with larger files.

Tips for Running Compaction

Target file size: Match your engine’s ideal scan size. 128MB or 256MB often work well.
Partition scope: You can compact per partition to avoid touching the entire table.
Job parallelism: Tune parallelism to handle large volumes efficiently.
Avoid overlap: If streaming ingestion is running, compaction jobs should avoid writing to the same partitions concurrently (we’ll cover this in Part 3).

When Should You Run It?

That depends on:

Ingestion frequency: Frequent writes = more small files = more frequent compaction
Query behavior: If queries touch recently ingested data, compact often
Table size and storage costs: The larger the table, the more benefit from compaction

In many cases, a daily or hourly schedule works well. Some platforms support event-driven compaction based on file count or size thresholds.

Tradeoffs

While compaction boosts performance, it also:

Consumes compute and I/O resources
Temporarily increases storage (until old files are expired)
Can interfere with concurrent writes if not carefully scheduled

That’s why timing and scope matter—a theme we’ll return to later in this series.

Up Next

Now that you understand standard compaction, the next challenge is applying it without interrupting streaming workloads. In Part 3, we’ll explore techniques to make compaction faster, safer, and more incremental for real-time pipelines.

The Cost of Neglect — How Apache Iceberg Tables Degrade Without Optimization

Alex Merced — Tue, 15 Jul 2025 09:00:00 GMT

The Cost of Neglect — How Apache Iceberg Tables Degrade Without Optimization

Apache Iceberg offers powerful features for managing large-scale datasets with reliability, versioning, and schema evolution. But like any robust system, Iceberg tables require care and maintenance. Without ongoing optimization, even the most well-designed Iceberg table can degrade—causing query slowdowns, ballooning metadata, and rising infrastructure costs.

This post kicks off a 10-part series on Apache Iceberg Table Optimization, beginning with a look at what happens when you don’t optimize and why it matters.

Why Do Iceberg Tables Degrade?

At its core, Iceberg uses a table metadata layer to track the location and structure of physical files (data files, manifests, and manifest lists). Over time, various ingestion patterns—batch loads, streaming micro-batches, late-arriving records—can lead to an accumulation of inefficiencies:

1. Small Files Problem

Each write operation typically creates a new data file. In streaming or frequent ingestion pipelines, this can lead to thousands of tiny files that:

Increase the number of file system operations during scans
Reduce the effectiveness of predicate pushdown and pruning
Add overhead to table metadata (larger manifest files)

2. Fragmented Manifests

Each new snapshot creates new manifest files. If the same files appear in many manifests or are not compacted, snapshot metadata becomes expensive to read and maintain.

3. Bloated Snapshots

Iceberg maintains a full history of table snapshots unless explicitly expired. Over time, this bloats the metadata layer with obsolete entries:

Slows down time travel and rollback operations
Inflates table size even if the data volume is static
Consumes storage and memory unnecessarily

4. Unclustered or Unsorted Data

Without explicit clustering or sort order, files may be written in a way that scatters relevant records across multiple files. This leads to:

Increased scan ranges and data reads during filtering
Poor locality for analytical queries

5. Partition Imbalance

When partitions grow at uneven rates, you may end up with:

Some partitions containing massive files
Others being overloaded with small files
Query planning bottlenecks on overgrown partitions

What Are the Consequences?

These degradations manifest as tangible issues across your data platform:

Performance Hits: Query scans take longer and use more compute resources.
Higher Costs: More files and metadata inflate cloud storage bills and increase query processing cost in engines like Dremio, Trino, or Spark.
Longer Maintenance Windows: Snapshot expiration, schema evolution, and compaction become more expensive over time.
Reduced Freshness and Responsiveness: Particularly in streaming use cases, lag builds up if optimizations are not happening incrementally.

What Causes This Degradation?

Most of these issues stem from a lack of:

Regular compaction
Snapshot and metadata cleanup
Monitoring table health metrics
Clustering and layout optimization during writes

Looking Ahead

The good news is that Apache Iceberg provides powerful tools to fix these issues—with the right strategy. In the next posts, we’ll break down each optimization method, starting with standard compaction and how to implement it effectively.

Stay tuned for Part 2: The Basics of Compaction — Bin Packing Your Data for Efficiency

How to Discover or Organize Lakehouse & Apache Iceberg Meetups

Alex Merced — Thu, 03 Jul 2025 09:00:00 GMT

Free Resources

Planning a meetup around Apache Iceberg or modern data lakehouse architectures? Whether you're looking to host your first community event or expand your existing network, discovering and organizing meetups can be both rewarding and impactful. These gatherings offer an opportunity to connect with other data professionals, share best practices, and explore cutting-edge tools and architectures. In this blog, we'll explore how to find and collaborate with existing data communities, discover upcoming Iceberg and lakehouse-related events, and provide tips on organizing your own meetup. We'll also share links to online communities, tools, and platforms to help you build momentum around your event and grow your local or virtual data community.

Step 1: Join the Related Communities

Slack communities for different lakehouse communities are going to be one of the best places to find people to collaborate with. In certain communities there are dedicated channels for meetups that make easier to discover people looking to collaborate in your area

Step 2: Where to collaborate

A good pattern to use is to create a meetup channel if it doesn't already exist for your area like #meetup-atlanta and then invite people to join the channel to collaborate on local meetups.

Data Events Slack Community

The Data Events Slack Community is a great place to find people to collaborate with. Here are the existing meetup channels in the Data Events Slack Community:

meetup-argentina
meetup-australia
meetup-brazil
meetup-california
meetup-canada
meetup-chile
meetup-china
meetup-colombia
meetup-colorado
meetup-egypt
meetup-florida
meetup-france
meetup-georgia
meetup-germany
meetup-illinois
meetup-india
meetup-ireland
meetup-israel
meetup-japan
meetup-massachusetts
meetup-mexico
meetup-netherlands
meetup-newyork
meetup-northcarolina
meetup-singapore
meetup-southafrica
meetup-southkorea
meetup-sweden
meetup-texas
meetup-uk
meetup-utah
meetup-washington

Iceberg Slack

Currently in the Apache Iceberg Slack Workspace the following Channels Exist:

meetup-atlanta
meetup-austin
meetup-bayarea
meetup-boston
meetup-chicago
meetup-denver
meetup-nola
meetup-orlando
meetup-seattle

note: There are no other channels for other cities as the ability to make channels was turned off in the Iceberg Slack, my suggestion is make the channel in the Data Lakehouse Hub slack.

Data Lakehouse Hub Slack

Here are the existing meetup channels in the Data Lakehouse Hub Slack:

meetup-atlanta
meetup-austin
meetup-barcelona
meetup-boston
meetup-chicago
meetup-denver
meetup-london
meetup-miami
meetup-munich
meetup-nyc
meetup-nola
meetup-orlando
meetup-san-francisco
meetup-santa-clara
meetup-seattle

Apache Polaris Slack

There is a #meetup-attendee and #meetup-organizer channel in the Apache Polaris Slack along with:

#meetup-nyc-austin-boston-atlanta
#meetup-sanfran-seattle-denver-chicago

Step 3: Find or Propose Events

By reading these channels you should be able to discover upcoming Iceberg and lakehouse-related events in your area. If you want to organize an event you can propose an event and see who would want to collaborate in organizing the event.

Step 4: Organize the Event

Naming Your Event

Simplest way to organize your event is under the name X Lakehouse Meetup where X is the city or region and you can run the meetup any way you like. For example, Atlanta Lakehouse Meetup. But if you want to use a name like Atlanta Apache Iceberg Meetup you can do that but need to follow recently approved guidelines for doing so to avoid trademark issues with the Apache Software Foundation.

Apache Iceberg should be championed in every meetup and technical session (after all, we're here to support this technology and our community)
All talks should be vendor-neutral and not sales pitches (of course vendors can be mentioned, but that should never be the point of the talk)
Each meetup should have at least two talks with speakers representing different companies/organizations (we need to champion diversity of thought)
Planned meetups ought to be brought to the attention of the dev list (this is to promote transparency and raise awareness)

These rules include having an open call for speakers prior to the event and decided on the speakers among all event sponsors (and allow others to sponsor the event if they want to).

Organizing the Event

Essentially you have three main costs:

Venue
Drinks
Food

So soliciting people to co-sponsor the event either by sharing the costs of these things or having different sponsorts pay for different things is a good way to organize the event.

All contriuting sponsors should have their logos on the event promotion. You'll want all these details squared away to allow at least 2 weeks of promotion before the event if not more.

Promoting the Event

You should first either create a meetup or lu.ma listing for the event. For Apache Iceberg meetups there are community run outlets to post your event.

Here are some other Luma Calendars and Meetup Groups you may want to follow for Lakehouse Events:

Meetup Groups

Luma Calendars

Message calendars@datalakehousehub.com to get your event added to these calendars, include link to Luma or Meetup event listing:

Social Media

Make sure everyone involved is posting about the event on linkedin, twitter and blue sky.

Emails

Sponsors should send emails about the event to their lists if they can, use Luma to email attendees to remind them about the event 7 days, 24 hours and 2 hours before the event with any logistics details they should know. Offering each sponsor a link in these emails to a related blog or asset is a good idea.

Conclusion

Bringing together the Lakehouse and Apache Iceberg community through meetups is one of the most effective ways to foster collaboration, share knowledge, and build meaningful relationships across organizations and regions. Whether you're organizing your first meetup or joining an existing one, the open and welcoming nature of these communities makes it easy to get involved. By leveraging platforms like Slack, Luma, and Meetup, and by following best practices for organizing inclusive and impactful events, you can help grow the ecosystem and play a key role in advancing open data architectures. So jump into a meetup channel, connect with others, and start planning — your community is waiting.

What is an API? And Why Data Architecture Depends on Them

Alex Merced — Mon, 23 Jun 2025 09:00:00 GMT

Free Resources

Imagine walking into a restaurant in a foreign country where you don’t speak the language. You point at things, gesture wildly, maybe even draw pictures — anything to communicate what you want. But if you and the server spoke a common language like English or Spanish, things would go a lot smoother.

That’s exactly what APIs do for software systems. They are shared languages that define how software components talk to each other. Without a shared API, systems can't collaborate easily, leading to miscommunication, friction, or total breakdown.

In this post, we'll unpack what APIs are and why they’re critical in data architecture. We'll explore the different types of APIs, how they've shaped modern data workflows, and the standards that have emerged in key areas like storage, data transport, and cataloging. Whether you're a developer building integrations or a data architect planning your stack, understanding these APIs is essential for navigating today's complex data ecosystem.

What is an API?

An API, or Application Programming Interface, is like a contract that defines how different software components can interact. Think of it as a language specification — if two programs speak the same API, they can communicate effectively, even if they're written in different languages or run on different platforms.

Just like a language has rules for grammar and vocabulary, an API defines the rules for how requests are made, what data is expected, and how responses are structured. When software follows these rules, integration becomes smooth and predictable.

It's important to recognize that the term "API" can mean different things depending on context:

In software development, an API can refer to the functions and methods exposed by a library or class. If one class implements the same method signatures as another, it can serve as a drop-in replacement.
In system integration, APIs more commonly refer to how different applications or services communicate over a network, especially using HTTP. This includes how data is sent, what endpoints exist, and how authentication is handled.

In essence, APIs enable modularity and collaboration in software. They allow teams to build components independently, knowing they can connect through a well-defined interface.

The Four Horsemen of HTTP APIs

When most people talk about APIs in modern software systems, they’re usually referring to HTTP-based APIs — interfaces that allow software to communicate over the web or internal networks. Over time, four main styles of HTTP APIs have emerged, each with its own strengths and trade-offs.

1. SOAP (Simple Object Access Protocol)

SOAP is a protocol-based API style that uses XML to encode messages and enforces strict standards for how messages are structured. It includes built-in specifications for things like security and error handling. While powerful, SOAP is often seen as heavyweight and complex, which has led to a decline in its use for most new applications.

2. REST (Representational State Transfer)

REST is more lightweight and flexible. It uses standard HTTP methods like GET, POST, PUT, and DELETE to perform operations on resources, which are identified via URLs. REST APIs are stateless, meaning each request contains all the information needed to process it. REST's simplicity and widespread adoption have made it the go-to style for many web services.

3. RPC (Remote Procedure Call)

RPC is all about invoking functions remotely. Instead of thinking in terms of resources, you think in terms of actions — like calling a method named getUserDetails. RPC can use different serialization formats (like JSON-RPC or gRPC) and tends to be more efficient for certain tasks, especially internal service communication.

4. GraphQL

GraphQL allows clients to request exactly the data they need and nothing more. Instead of multiple endpoints, there’s typically a single endpoint that interprets a query language. This can reduce over-fetching and under-fetching of data and provides a more dynamic interface, especially useful for frontend applications.

Each of these API types has its place in the ecosystem. Understanding their differences helps you pick the right tool for the job depending on complexity, flexibility, and performance needs.

Why APIs Matter in Modern Data Architecture

The modern data stack is a vibrant and diverse ecosystem. From ingestion tools and storage layers to transformation engines and visualization platforms, each component often comes from a different vendor or open-source project. The glue that holds this ecosystem together is the API.

With so many tools available, the ability to integrate them seamlessly becomes a competitive advantage. Instead of reinventing the wheel, software platforms that adopt well-known APIs can plug into existing workflows and leverage established tooling. This interoperability allows teams to mix and match components without being locked into a single vendor or technology stack.

For example, if two different tools both understand the same API for reading from a data catalog or writing to object storage, they can work together out of the box. This eliminates the need for custom connectors or fragile workarounds.

APIs also encourage specialization. A tool can focus on doing one thing well — like cataloging metadata or transporting data — and expose an API that others can build upon. This modularity is what makes today's data architectures more flexible and scalable than ever before.

In short, APIs are the foundation of composability in data systems. They allow different parts of the stack to evolve independently while still working together in harmony.

Case Study – The Ubiquity of the S3 API

Amazon S3 wasn't just a game changer because it offered scalable cloud storage. It also introduced a clean, consistent API that made storing and retrieving objects over the web straightforward. This API became so widely adopted that it evolved into a de facto standard for cloud object storage.

As other cloud providers and storage platforms emerged, they faced a choice: create their own APIs or adopt the S3 API. Many chose the latter. Why? Because the S3 API already had a massive ecosystem of integrations. Backup tools, data lakes, ETL pipelines, and analytics platforms already knew how to talk to S3. By supporting the S3 API, new storage services could plug into these tools without requiring any custom development.

This is a powerful example of how API adoption fuels interoperability. Instead of forcing users to learn a new interface or rebuild their workflows, S3-compatible services ride the wave of existing infrastructure. As a result, users get flexibility and choice without sacrificing compatibility.

The takeaway: when an API reaches critical mass, it becomes more than a technical interface — it becomes an ecosystem enabler.

Data Transport APIs – From JDBC/ODBC to ADBC

Moving data between systems has always been a core challenge in data architecture. For decades, the standard approach involved using JDBC (Java Database Connectivity) and ODBC (Open Database Connectivity). These APIs allowed applications to connect to relational databases in a consistent way, abstracting the underlying database-specific protocols.

While JDBC and ODBC have served well, they come with limitations. These APIs were designed for transactional systems and row-based data access. As analytics workloads became more complex and data volumes grew, these traditional interfaces began to show performance bottlenecks.

It’s also important to note that JDBC and ODBC are not HTTP-based APIs. They operate over lower-level network protocols tailored to database drivers and client libraries. This can make them harder to integrate in cloud-native or language-agnostic environments.

Enter ADBC (Arrow Database Connectivity), a modern alternative designed for analytical use cases. ADBC builds on Arrow Flight, which is a gRPC-based protocol optimized for high-throughput data transport. Instead of transferring rows one by one, Arrow Flight sends columnar batches over a persistent connection, dramatically improving efficiency for analytical queries.

With ADBC, the API is designed for today’s needs: fast, language-agnostic, and cloud-friendly. It embraces open standards like Apache Arrow and gRPC to deliver performance without sacrificing interoperability.

As analytics platforms grow more distributed and data-hungry, APIs like ADBC represent a forward-looking approach to data transport — one that matches the scale and speed of modern data systems.

Data Catalog APIs – Hive, Glue, and Iceberg REST

Lakehouse Data catalogs store metadata about datasets — such as schema, location, and partitioning — allowing tools to discover and manage data assets consistently. But for this ecosystem to function, catalogs need APIs that other tools can understand.

Three primary catalog APIs have emerged in the lakehouse and analytics space:

1. Hive Metastore API

The Hive API was one of the earliest standards for metadata management in Hadoop-based systems. Because Apache Hive gained significant adoption early on, its metastore API became widely supported. Even tools that don’t use Hive for querying often support its API for interoperability.

2. AWS Glue Catalog API

As AWS became a dominant platform for cloud-native analytics, its Glue Catalog gained traction. Glue offered a managed alternative to Hive with cloud-native scalability and tight integration with AWS services. Many tools added support for Glue to integrate seamlessly within AWS ecosystems.

3. Apache Iceberg REST Catalog API

The Iceberg project initially struggled with catalog integration due to varying implementations. To solve this, the community introduced a REST-based catalog API that standardizes how tools interact with Iceberg catalogs regardless of the underlying backend. This REST interface provides a clear contract and enables broader compatibility. Catalogs that support the Iceberg REST Catalog (IRC) API include Apache Polaris (incubating), Apache Gravitino, Dremio Catalog, Open Catalog, AWS Glue Catalog, Lakekeeper, Nessie, Unity Catalog and many more. Most specialized Iceberg tooling uses this as the main catalog API for discovering your Apache Iceberg datasets while catalogs like Polaris, Gravitino and Unity also adopt other APIs to make additional datasets discoverable.

Today, most lakehouse tools support one or more of these APIs to ensure compatibility across different environments. Whether you're working with on-prem systems using Hive, cloud-native stacks using Glue, or modern lakehouse engines built around Iceberg, API adoption remains the key to ecosystem integration.

Choosing catalog tools that support these APIs ensures you're building on a foundation that promotes interoperability, flexibility, and future-proofing.

Conclusion

APIs are more than just technical interfaces — they are the connective tissue of modern software. In data architecture, where tools span a wide range of functions and vendors, APIs enable these components to work together smoothly.

We’ve seen how APIs act like shared languages, allowing software to communicate efficiently. From foundational HTTP-based APIs like REST and GraphQL, to specialized data interfaces like the S3 API, JDBC, ADBC, and various catalog APIs, each plays a role in shaping the data landscape.

By adopting established APIs, tools become more compatible, easier to integrate, and more valuable within the broader ecosystem. And for data teams, aligning on common APIs means less time wrestling with custom connectors and more time delivering insights.

As the data world continues to evolve, understanding and leveraging key APIs is essential. They’re not just part of the plumbing — they’re a strategic asset for building robust, scalable, and flexible data systems.

Decoding AWS EC2 Instance Type Names

Alex Merced — Wed, 18 Jun 2025 09:00:00 GMT

Free Resources

Introduction

If you've ever browsed AWS EC2 instance types and found yourself staring blankly at names like m5.large, c6g.xlarge, or r7a.2xlarge, you're not alone. At first glance, these names can feel cryptic—like trying to decode a secret code.

But here's the good news: there's a method to the madness. Each part of an instance type name tells you something important about the underlying hardware, performance characteristics, and intended use case.

In this blog post, we'll break down the structure of AWS instance type names and show you how to read them like a pro. Once you understand how to interpret each component, you'll be able to confidently choose the right instance for your workload—and maybe even impress your colleagues with your cloud fluency.

The Anatomy of an Instance Type

Every AWS EC2 instance type name is composed of distinct parts that reveal critical details about the instance's capabilities. The general structure looks like this:

[family][generation][optional suffix].[size]

Let’s take the instance type c6g.large as an example:

c → Compute optimized
6 → 6th generation hardware
g → Powered by AWS Graviton (ARM-based processor)
large → Medium-sized instance (typically 2 vCPUs and 4 GB RAM)

By understanding what each segment means, you can quickly assess whether an instance is optimized for compute, memory, storage, or GPU, and how big or powerful it is.

In the sections below, we’ll walk through each part of the name in more detail.

Family – What Is the Instance Optimized For?

The first letter (or set of letters) in an instance type indicates the instance family, which tells you what the instance is optimized for. This helps guide your choice based on the nature of your workload—whether you need general-purpose performance, high CPU, large memory, or GPU acceleration.

Here’s a quick overview of the most common instance families:

Family	Description	Common Use Cases
`t`	Burstable general purpose	Development, low-traffic websites
`m`	General purpose	Balanced CPU and memory workloads
`c`	Compute optimized	High-performance computing, batch processing
`r`	Memory optimized	In-memory databases, real-time analytics
`x`	Extra memory optimized	SAP HANA, memory-intensive enterprise apps
`i`	Storage optimized (high IOPS)	NoSQL databases, large transactional systems
`g`	GPU instances	Machine learning, video rendering
`p`	High-performance GPU	Deep learning training, scientific modeling
`h`, `d`, `z`	Specialized families	Varies (HPC, local storage, high-frequency)

Understanding the family is the first step in selecting the right instance. For example, if your application is CPU-bound, a c family instance will typically deliver better performance per dollar than an m or t instance.

Generation – How New Is the Hardware?

The number immediately following the family letter represents the generation of the instance. AWS continuously improves its infrastructure, and newer generations typically offer better performance, energy efficiency, and cost-effectiveness compared to older ones.

For example:

m4 → 4th generation general-purpose instance
m5 → Newer 5th generation version
m6g → 6th generation with Graviton (ARM-based processor)

Why It Matters:

Choosing a newer generation instance usually means access to:

Improved CPUs (e.g., Intel Ice Lake, AMD EPYC, or AWS Graviton)
Better network and storage throughput
Lower cost for similar or better performance

That said, not all regions have the latest generation available. Always check your region’s instance offerings and benchmark critical workloads if performance is a top priority.

Suffix – Special Chips or Capabilities

Some instance types include an optional suffix—a letter (or combination of letters) that provides additional detail about the instance’s hardware or features. These suffixes appear immediately after the generation number and can help you identify special variants optimized for particular use cases.

Common Suffixes and What They Mean:

Suffix	Meaning	Description
`a`	AMD EPYC processor	Cost-effective alternative to Intel-based instances
`g`	AWS Graviton processor (ARM-based)	Energy-efficient, high performance, lower cost
`n`	Network-optimized	Enhanced network bandwidth and performance
`d`	Includes local NVMe storage	Fast local instance storage for low-latency workloads
`e`	Extended memory or enhanced features	More memory or improved capabilities per vCPU
`z`	High-frequency Intel CPUs	For workloads that need very high clock speed

Example:

r6a → Memory optimized (r), 6th generation, AMD processor (a)
m6g → General purpose (m), 6th generation, Graviton processor (g)
i3d → Storage optimized (i), 3rd generation, with NVMe instance store (d)

These suffixes allow you to fine-tune your instance selection based on price, performance, or architecture preferences—especially important if your software is architecture-sensitive (e.g., x86 vs ARM).

Size – How Big Is the Instance?

The part of the instance type that comes after the period (.) defines the size of the instance. This determines how many vCPUs, how much memory, and sometimes how much networking or storage bandwidth is allocated.

AWS uses consistent naming for sizes across instance families:

Size	Description	Typical vCPUs	Notes
`.nano`	Very small	1	For ultra-light workloads
`.micro`	Small	1	Entry-level, burstable performance
`.small`	Modest	1–2	Slightly more consistent CPU
`.medium`	Standard	1–2	Balanced for small apps
`.large`	2x baseline	2	Common for dev/test workloads
`.xlarge`	4x baseline	4	Heavier compute or memory needs
`.2xlarge`	8x baseline	8	Medium to large production loads
`.4xlarge`	16x baseline	16	High-capacity apps
`.8xlarge`	32x baseline	32	Data processing, analytics
`.12xlarge`	48x baseline	48	High-scale enterprise workloads
`.24xlarge`	96x baseline	96	Very high-performance computing
`.metal`	Bare metal (no hypervisor)	Varies	Full access to physical server

Example:

m5.large = General-purpose instance, 5th generation, with 2 vCPUs and 8 GB memory.
c6g.4xlarge = Compute optimized, 6th gen, Graviton processor, with 16 vCPUs and 32 GB memory.

Choosing the right size allows you to scale vertically by increasing resources within a single instance, or horizontally by adding more instances of a smaller size depending on your architecture and cost goals.

Pulling It All Together

Now that you understand each component—family, generation, suffix, and size—you can decode any EC2 instance type and understand exactly what it offers.

Let’s break down a few examples to reinforce what you’ve learned:

🔹 Example 1: `c6g.large`

c → Compute optimized
6 → 6th generation
g → AWS Graviton (ARM-based processor)
large → Medium-sized (2 vCPUs, ~4 GB RAM)

Use case: Great for compute-heavy applications running on ARM, like containerized services or microservices at scale.

🔹 Example 2: `r5d.4xlarge`

r → Memory optimized
5 → 5th generation
d → Includes local NVMe SSD instance store
4xlarge → 16 vCPUs and 128 GB RAM

Use case: Ideal for high-throughput, in-memory databases or data processing that benefits from fast local storage.

🔹 Example 3: `m7a.xlarge`

m → General purpose
7 → 7th generation
a → AMD EPYC processor
xlarge → 4 vCPUs, 16 GB RAM

Use case: Balanced workloads where cost-effectiveness is important, such as web applications or business logic layers.

Understanding how to read these names makes it easier to compare instance types, choose the best fit for your application, and avoid over-provisioning. You’ll save money, optimize performance, and build with more confidence on AWS.

Introduction to Data Engineering Concepts | What is Data Engineering?

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

Data engineering sits at the heart of modern data-driven organizations. While data science often grabs headlines with predictive models and AI, it's the data engineer who builds and maintains the infrastructure that makes all of that possible. In this first post of our series, we’ll explore what data engineering is, why it matters, and how it fits into the broader data ecosystem.

The Role of the Data Engineer

Think of a data engineer as the architect and builder of the data highways. These professionals design, construct, and maintain systems that move, transform, and store data efficiently. Their job is to ensure that data flows from various sources into data warehouses or lakes where it can be used reliably for analysis, reporting, and machine learning.

In a practical sense, this means working with pipelines that connect everything from transactional databases and API feeds to large-scale storage systems. Data engineers work closely with data analysts, scientists, and platform teams to ensure the data is clean, consistent, and available when needed.

From Raw to Refined: The Journey of Data

Raw data is rarely useful as-is. It often arrives incomplete, messy, or inconsistently formatted. Data engineers are responsible for shepherding this raw material through a series of processing stages to prepare it for consumption.

This involves tasks like:

Data ingestion (bringing data in from various sources)
Data transformation (cleaning, enriching, and reshaping the data)
Data storage (choosing optimal formats and storage solutions)
Data delivery (ensuring end users can access data quickly and easily)

At each stage, considerations around scalability, performance, security, and governance come into play.

Data Engineering vs Data Science

It's common to see some confusion between the roles of data engineers and data scientists. While their work is often complementary, their responsibilities are distinct.

A data scientist focuses on analyzing data and building predictive models. Their tools often include Python, R, and statistical frameworks. On the other hand, data engineers build the systems that make the data usable in the first place. They are often more focused on infrastructure, system design, and optimization.

In short: the data scientist asks questions; the data engineer ensures the data is ready to answer them.

A Brief History of the Data Stack

The evolution of data engineering can be seen in how the data stack has changed over time.

In traditional environments, organizations relied heavily on ETL tools to move data from relational databases into on-premise warehouses. These systems were tightly controlled but not particularly flexible or scalable.

With the rise of big data, open-source tools like Hadoop and Spark introduced new ways to process data at scale. More recently, cloud-native services and modern orchestration frameworks have enabled even more agility and scalability in data workflows.

This evolution has led to concepts like the modern data stack and data lakehouse—topics we’ll cover later in this series.

Why It Matters

Every modern organization depends on data. But without a solid foundation, data becomes a liability rather than an asset. Poorly managed data can lead to flawed insights, compliance issues, and lost opportunities.

Good data engineering practices ensure that data is:

Accurate and timely
Secure and compliant
Scalable and performant

In a world where data volumes and velocity are only increasing, the importance of data engineering will only continue to grow.

What’s Next

Now that we’ve outlined the role and importance of data engineering, the next step is to explore how data gets into a system in the first place. In the next post, we’ll dig into data sources and the ingestion process—how data flows from the outside world into your ecosystem.

Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

Before we can analyze, model, or visualize data, we first need to get it into our systems. This step—often taken for granted—is known as data ingestion. It’s the bridge between the outside world and the internal data infrastructure, and it plays a critical role in how data is shaped from day one.

In this post, we’ll break down the types of data sources you’ll encounter, the ingestion strategies available, and what trade-offs to consider when designing ingestion workflows.

What Are Data Sources?

At its core, a data source is any origin point from which data can be extracted. These sources vary widely in structure, velocity, and complexity.

Relational databases like MySQL or PostgreSQL are common sources in transactional systems. They tend to produce highly structured, row-based data and are often central to business operations such as order processing or customer management.

APIs are another rich source of data, especially in modern SaaS environments. From financial data to social media feeds, APIs expose endpoints where structured (often JSON-formatted) data can be requested in real-time or on a schedule.

Then there are flat files—CSV, JSON, XML—often used in data exports, logs, and external data sharing. While simple, they can carry critical context or fill gaps that structured sources miss.

Sensor data, clickstreams, mobile apps, third-party tools, and message queues all add to the landscape, each bringing its own cadence and complexity.

Ingestion Strategies: Batch vs Streaming

Once you identify your sources, the next question becomes: how will you ingest the data?

Batch ingestion involves collecting data at intervals and processing it in chunks. This could be once a day, every hour, or even every minute. It's suitable for systems that don't require real-time updates and where data can afford to be a little stale. For example, nightly financial reports or end-of-day sales data.

Batch processes tend to be simpler and easier to maintain. They can rely on traditional extract-transform-load (ETL) workflows and are often orchestrated using tools like Apache Airflow or simple cron jobs.

Streaming ingestion, on the other hand, handles data in motion. As new records are created—say, a customer clicks a link or a sensor detects a temperature change—they’re ingested immediately. This method is crucial for use cases that require low-latency or real-time processing, such as fraud detection or live recommendation engines.

Apache Kafka is a popular tool for enabling streaming pipelines. It allows systems to publish and subscribe to streams of records, ensuring data flows continuously with minimal delay.

Structured, Semi-Structured, and Unstructured Data

Understanding the shape of your data also influences how you ingest it.

Structured data is highly organized and fits neatly into tables. Think SQL databases or CSV files. Ingestion here often involves direct connections via JDBC drivers, SQL queries, or file uploads.

Semi-structured data, like JSON or XML, has an internal structure but doesn’t conform strictly to relational models. Ingesting this data may require parsing logic and schema inference before it's usable downstream.

Unstructured data includes images, videos, PDFs, and raw text. These formats typically require specialized tools and more complex handling, often involving metadata extraction or integration with machine learning models for classification or tagging.

Considerations in Designing Ingestion Pipelines

Data ingestion isn’t just about moving bytes—it’s about doing so reliably, efficiently, and with the future in mind.

Latency requirements play a major role. Does the business need data as it happens, or is yesterday’s data good enough? That determines your choice between batch and streaming.

Scalability is another concern. What works for 10,000 records a day might break under 10 million. Tools like Kafka and cloud-native services such as AWS Kinesis or Google Pub/Sub help handle high throughput without compromising performance.

Error handling is essential. What happens if a source API goes down? What if a file arrives with missing fields? Designing retry logic, alerts, and fallback mechanisms helps ensure ingestion pipelines are robust.

Finally, schema evolution can’t be overlooked. Data changes over time—columns get added, data types shift. Your ingestion pipeline must be flexible enough to adapt without breaking downstream systems.

Looking Ahead

Getting data into the system is just the beginning. Once it’s ingested, it often needs to be transformed to fit the analytical or business context.

In the next post, we’ll explore the concepts of ETL and ELT—two core paradigms for moving and transforming data—and look at how they differ in practice and purpose.

Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

Once data has been ingested into your system, the next step is to prepare it for actual use. This typically involves cleaning, transforming, and storing the data in a way that supports analysis, reporting, or further processing. This is where data pipelines come in, and at the center of pipeline design are two common strategies: ETL and ELT.

Although they may look similar at first glance, ETL and ELT represent fundamentally different approaches to handling data transformations, and each has its strengths and trade-offs depending on the context in which it’s used.

What is ETL?

ETL stands for Extract, Transform, Load. It’s the traditional method used in many enterprise environments for years. The process starts by extracting data from source systems such as databases, APIs, or flat files. This raw data is then transformed—typically on a separate processing server or ETL engine—before it is finally loaded into a data warehouse or other destination system.

For example, imagine a retail company collecting daily sales data from multiple stores. In an ETL workflow, the system might extract those records at the end of the day, standardize formats, filter out corrupted rows, aggregate sales by region, and then load the clean, transformed dataset into a reporting warehouse like Snowflake or Redshift.

One of the key advantages of ETL is that it allows you to load only clean, verified data into your warehouse. That often means smaller storage footprints and potentially better performance on downstream queries.

However, this approach also has limitations. Because the transformation happens before loading, you must decide upfront how the data should be shaped. If business rules change or additional use cases emerge, you may need to go back and reprocess the data.

What is ELT?

ELT reverses the order of the last two steps: Extract, Load, Transform. In this model, raw data is extracted from the source and immediately loaded into the target system—usually a cloud data warehouse that can scale horizontally. Once the data is in place, transformations are performed within the warehouse using SQL or warehouse-native tools.

This approach takes advantage of the high compute power and scalability of modern cloud platforms. Instead of bottlenecking on a dedicated ETL server, the warehouse can handle complex joins, aggregations, and transformations at scale.

Let’s go back to the retail example. With ELT, all sales data is loaded as-is into the warehouse. Analysts or data engineers can then write transformation scripts to reshape the data for various use cases—trend analysis, regional comparisons, or fraud detection—all without having to re-ingest or reload the source data.

ELT offers more flexibility for evolving requirements, supports broader self-service analytics, and enables faster time-to-insight. The trade-off is that it requires strong governance and monitoring. Because raw data is stored in the warehouse, the risk of exposing inconsistent or unclean data is higher if transformation logic isn’t managed carefully.

Choosing Between ETL and ELT

The decision to use ETL or ELT often depends on your stack, performance needs, and organizational practices.

ETL still makes sense in environments with strict data governance, limited warehouse compute resources, or scenarios where only clean data should be retained. It’s also common in legacy systems and on-premise architectures.

ELT shines in modern cloud-native environments where scalability and agility are top priorities. It’s often used with platforms like Snowflake, BigQuery, or Redshift, which are built to handle large volumes of raw data and complex SQL-based transformations efficiently.

In practice, many organizations use a hybrid approach. Critical data may go through an ETL flow, while experimental or rapidly evolving datasets follow an ELT pattern.

The Bigger Picture

ETL and ELT are just different roads to the same destination: getting data ready for use. As the modern data stack evolves, so do the tools and best practices for managing these flows. Whether you choose one approach or blend both, what matters most is building pipelines that are reliable, maintainable, and aligned with your organization’s goals.

In the next post, we’ll focus on batch processing—the traditional foundation of many ETL workflows—and discuss how data engineers design, schedule, and optimize these processes for scale.

Introduction to Data Engineering Concepts | Batch Processing Fundamentals

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

For many data engineering tasks, real-time insights aren’t necessary. In fact, a large portion of the data processed across organizations happens in scheduled intervals—daily sales reports, weekly data refreshes, monthly billing cycles. This is where batch processing comes in, and despite the growing popularity of streaming, batch remains the backbone of many data-driven workflows.

In this post, we’ll explore what batch processing is, how it works under the hood, and why it’s still a critical technique in the data engineer’s toolbox.

What is Batch Processing?

Batch processing is the execution of data workflows on a predefined schedule or in response to specific triggers. Instead of processing data as it arrives, the system collects a set of data over a period of time, then processes that set as a single unit.

This approach is particularly useful when data arrives in large quantities but doesn’t need to be acted on immediately. For example, processing daily transactions from a point-of-sale system or generating overnight reports for executive dashboards.

Batch jobs are often triggered at set times—say, every night at 2 a.m.—and are designed to run until completion, often without user interaction. They can run for seconds, minutes, or even hours depending on the volume of data and complexity of the transformations.

Under the Hood: How Batch Jobs Work

The anatomy of a batch job usually includes several stages. First, the job identifies the data it needs to process. This might involve querying a database for all records created in the last 24 hours or scanning a specific folder in object storage for new files.

Next comes the transformation phase. This is where data is cleaned, filtered, joined with other datasets, and reshaped to fit its target structure. This phase can include tasks like date formatting, currency conversion, null value imputation, or the calculation of derived fields.

Finally, the job writes the transformed data to its destination—often a data warehouse, data lake, or downstream reporting system.

To manage all of this, engineers rely on workflow orchestration tools. These tools provide scheduling, error handling, and logging capabilities to ensure that jobs run in the right order and can recover gracefully from failure.

Tools and Technologies

Several tools have become staples in batch-oriented workflows. Apache Airflow is one of the most widely used. It allows engineers to define complex workflows as Directed Acyclic Graphs (DAGs), where each node represents a task and dependencies are explicitly declared.

Other tools like Luigi and Oozie offer similar functionality, though they are less commonly used in newer stacks. Cloud-native platforms such as AWS Glue and Google Cloud Composer provide managed orchestration services that integrate tightly with the respective cloud ecosystems.

In addition to orchestration, batch jobs often depend on distributed processing engines like Apache Spark. Spark allows massive datasets to be processed in parallel across a cluster of machines, reducing processing times dramatically compared to traditional single-node tools.

Strengths and Limitations

One of the biggest advantages of batch processing is its simplicity. Since data is processed in chunks, you can apply robust validation and error-handling routines before moving data downstream. It's also easier to track and audit, which is especially important for regulated industries.

Batch jobs are also cost-efficient when working with large volumes of data that don’t require immediate availability. Processing once per day means you can spin up compute resources only when needed, rather than keeping systems running continuously.

However, the main limitation is latency. If something happens in your business—say, a spike in fraudulent transactions—you won’t know about it until after the next batch job runs. For use cases that require faster insights or real-time responsiveness, batch processing isn’t sufficient.

There’s also the issue of windowing and completeness. Since batch jobs process data in slices, late-arriving records can fall outside the intended window unless carefully managed. This adds complexity to pipeline design and requires thoughtful handling of time-based logic.

Where Batch Still Shines

Despite its limitations, batch processing remains ideal for a wide range of use cases. Financial reconciliations, data archival, slow-changing dimensional data updates, and long-running analytics workloads are just a few examples where batch continues to dominate.

As a data engineer, understanding how to design efficient and reliable batch workflows is an essential skill, especially in environments where consistency and auditability are critical.

In the next post, we’ll explore the counterpart to batch: streaming data processing. We’ll look at what it means to process data in real time, how it differs from batch, and what patterns and tools make it work.

Introduction to Data Engineering Concepts | Streaming Data Fundamentals

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

In contrast to batch processing, where data is collected and processed in chunks, streaming data processing deals with data in motion. Instead of waiting for data to accumulate before running transformations, streaming pipelines ingest and process each piece of data as it arrives. This model enables organizations to respond to events in real time, a capability that’s becoming increasingly essential in domains like finance, security, and customer experience.

In this post, we’ll unpack the core ideas behind streaming, how it works in practice, and the challenges it presents compared to traditional batch systems.

What is Streaming Data?

Streaming data refers to data that is continuously generated by various sources—website clicks, IoT sensors, user interactions, system logs—and transmitted in real time or near-real time. This data typically arrives in small payloads, often as individual events, and needs to be processed with minimal delay.

The goal of a streaming pipeline is to capture this data as it’s generated, perform necessary transformations, and deliver it to its destination with as little latency as possible.

A simple example would be a ride-sharing app that tracks vehicle locations in real time. As each car moves, GPS data is streamed to a backend system that updates the user interface and helps dispatch rides based on current conditions.

How Streaming Systems Work

Unlike batch jobs that execute on a schedule, streaming systems run continuously. They consume data from a source, process it incrementally, and push it to a sink—all without waiting for a dataset to be complete.

At the heart of a streaming system is a message broker or event queue, which acts as a buffer between data producers and consumers. Apache Kafka is a popular choice here. It allows producers to publish events to topics, and consumers to read from those topics independently, often with strong guarantees around ordering and durability.

Once events are ingested, a processing engine takes over. Tools like Apache Flink, Spark Structured Streaming, and Apache Beam allow developers to apply transformations on a per-record basis or over time-based windows. This is where operations like filtering, aggregating, joining, and enriching occur.

These transformations must be designed to handle data that may arrive late, out of order, or in bursts. As such, streaming systems often implement complex logic to manage time—distinguishing between event time (when the event occurred) and processing time (when it was received)—to ensure results are accurate.

Use Cases and Business Impact

The appeal of streaming pipelines lies in their ability to power real-time applications. Fraud detection systems can flag suspicious transactions as they happen. E-commerce platforms can recommend products based on live browsing behavior. Logistics companies can monitor fleet activity and adjust routes on the fly.

In operational analytics, dashboards fed by streaming data provide up-to-the-minute visibility, allowing teams to make informed decisions in response to changing conditions.

Streaming is also a foundational component of event-driven architectures. When services communicate via events, streaming systems act as the glue that ties the application together, enabling asynchronous, decoupled interactions.

Challenges in Streaming Systems

Despite its power, streaming introduces complexity that shouldn’t be underestimated. Handling late or out-of-order data is a major concern. If an event shows up ten minutes after it was supposed to be processed, the system must be smart enough to either incorporate it correctly or account for the gap.

State management is another critical factor. When a pipeline needs to remember information across multiple events—like keeping a running total or maintaining a session—it must manage that state reliably, often across distributed systems.

There’s also the issue of fault tolerance. Streaming systems must be able to recover from crashes or network issues without duplicating results or losing data. This requires sophisticated checkpointing, replay, and exactly-once processing semantics, which tools like Flink and Beam are designed to provide.

Finally, testing and debugging streaming pipelines can be more difficult than batch jobs. Because they run continuously and deal with time-sensitive data, reproducing issues often requires specialized tooling or replay mechanisms.

When to Choose Streaming

Streaming makes sense when low-latency data processing is essential to the business. This could mean operational decision-making, customer experience personalization, or complex event processing in a microservices architecture.

It’s not always the right tool for the job, though. For workloads that don’t require immediate insights—or where simplicity and reliability matter more—batch processing remains the better choice.

As data engineers, the key is to understand the trade-offs and choose the right pattern for each use case.

In the next post, we’ll shift gears and look at how data is modeled for analytics. Understanding the differences between OLTP and OLAP systems, as well as the pros and cons of different schema designs, is critical to building pipelines that serve real business needs.

Introduction to Data Engineering Concepts | Data Modeling Basics

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

Behind every useful dashboard or analytics report lies a well-structured data model. Data modeling is the practice of shaping data into organized structures that are easy to query, analyze, and maintain. While it may sound abstract, modeling directly impacts how quickly and accurately data consumers can extract value from the information stored in your systems.

In this post, we’ll look at the foundations of data modeling, the difference between OLTP and OLAP systems, and common schema designs that data engineers use to build efficient and scalable data platforms.

Why Data Modeling Matters

When data arrives from source systems, it’s often raw and optimized for transactions, not analysis. A transactional database might record every sale or click in granular detail, but that structure doesn’t translate well into aggregations like “monthly revenue by product category.”

A data model reshapes that data to make it usable. Good models reduce complexity, improve performance, and minimize errors. Poor models, on the other hand, lead to slow queries, redundant data, and confusion about what numbers really mean.

Modeling is both a technical and a collaborative process. It requires not just understanding how data is structured, but also how the business thinks about that data—what questions need answering, how metrics are defined, and what trade-offs are acceptable.

OLTP vs OLAP: Two Worlds, Two Purposes

Before diving into specific modeling techniques, it’s important to distinguish between the two main types of data systems: OLTP and OLAP.

OLTP (Online Transaction Processing) systems are built for real-time operations. Think of point-of-sale systems, user authentication services, or banking apps. These systems are optimized for high-throughput reads and writes, handling thousands of small transactions per second. Their schemas are typically highly normalized to avoid data duplication and to keep updates fast and consistent.

OLAP (Online Analytical Processing) systems, on the other hand, are designed for analysis. These platforms support complex queries over large volumes of historical data. Performance here is about aggregating, filtering, and summarizing—not handling rapid transactions. Because of this, OLAP models often trade strict normalization for faster access to pre-joined or denormalized data.

Understanding whether your system is OLTP or OLAP helps determine how you model your data. The techniques and trade-offs are different depending on the system’s purpose.

Normalization and Denormalization

In OLTP systems, normalization is the standard. This means structuring data so that each fact is stored in exactly one place. For example, instead of storing a customer’s name with every order record, you keep customer details in a separate table and reference them via a key.

This approach minimizes redundancy, reduces storage, and simplifies updates. Change the customer’s name in one place, and every order reflects that change immediately.

In analytical systems, this level of indirection becomes a performance bottleneck. Complex queries must join many tables together, which can slow things down significantly.

That’s where denormalization comes in. In OLAP models, it’s common to store data in a flattened format, with descriptive attributes repeated across rows. While this increases storage requirements, it significantly speeds up query performance and simplifies logic for analysts and BI tools.

Star and Snowflake Schemas

Two common modeling patterns in OLAP systems are the star schema and the snowflake schema.

A star schema organizes data around a central fact table. This table holds measurable events—like sales transactions—with keys that reference surrounding dimension tables, which contain descriptive attributes such as product names, customer demographics, or store locations.

In a star schema, the dimension tables are typically denormalized. This makes queries straightforward and fast: one central join connects the fact table to all the attributes needed for analysis.

The snowflake schema takes this idea further by normalizing the dimension tables. Instead of a single product dimension table, for example, you might have separate tables for product, category, and supplier. This saves space and can improve maintainability, but at the cost of more complex joins.

The choice between star and snowflake schemas depends on your performance needs, data volume, and how often attributes change.

Modeling for Flexibility and Growth

Good data models are designed with change in mind. New columns will be added, relationships will evolve, and new metrics will be needed. A rigid model can become a bottleneck, while a flexible one supports ongoing development.

One best practice is to favor additive metrics when possible. These are measures you can safely sum across time or groups—like revenue or quantity sold. Additive metrics work better with aggregations and are easier to model consistently.

It’s also important to consider slowly changing dimensions. For example, if a customer’s email address or a product’s price changes, do you want to reflect the latest value, or keep historical versions? Modeling for this kind of change requires thought about versioning and historical accuracy.

The Road Ahead

Data modeling sits at the intersection of technical design and business logic. It’s not just about tables and keys—it’s about making data intuitive and useful for the people who depend on it.

As data engineers, our role is to create models that strike a balance between performance, maintainability, and expressiveness. Doing this well requires not just technical skill, but ongoing communication with analysts, stakeholders, and subject matter experts.

In the next post, we’ll take a closer look at data warehousing—how these models are stored, queried, and optimized in systems built for analytics at scale.

Introduction to Data Engineering Concepts | Data Warehousing Fundamentals

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

Data warehouses serve as the analytical backbone for many organizations. They are purpose-built systems that store structured data optimized for fast querying and aggregation. While data lakes handle raw, unstructured data at scale, data warehouses focus on delivering clean, organized datasets to analysts, BI tools, and decision-makers.

In this post, we'll break down what makes a data warehouse different from other storage systems, how it's architected, and what practices ensure it performs efficiently as your data and business grow.

The Role of a Data Warehouse

At a high level, a data warehouse collects data from multiple operational systems and stores it in a way that makes analysis easy and consistent. Instead of digging through individual source systems—like sales platforms, CRM tools, or web analytics—users can query a centralized warehouse that’s been curated and modeled for insight.

This consolidation allows organizations to apply consistent definitions for metrics, reduce the risk of conflicting data interpretations, and dramatically improve performance for analytical workloads.

Where a transactional database is designed to handle lots of small, rapid reads and writes, a data warehouse is designed to scan large volumes of data efficiently. These systems optimize for queries like “What were our top five products last quarter?” or “How did regional sales trend year-over-year?”

Architecture and Components

A traditional data warehouse is structured with a clear separation between compute and storage. In legacy on-premise systems like Teradata or Oracle, both functions were tightly coupled. In modern cloud-native systems like Snowflake or BigQuery, storage and compute are decoupled, which allows more flexible scaling.

The core of a warehouse is the schema—the logical structure defining how data is organized into tables, relationships, and hierarchies. As discussed in the previous post, these tables often follow star or snowflake patterns, with fact tables surrounded by dimension tables that provide context.

One of the key components of a warehouse is its query engine. This engine is built to efficiently execute SQL queries, taking advantage of indexing, partitioning, and columnar storage formats to return results quickly even when scanning billions of rows.

Data warehouses also maintain metadata—information about data types, table relationships, and data lineage—that helps users navigate and trust the system. Many modern platforms also offer built-in tools for access control, versioning, and data classification to support governance.

Performance Optimization: Partitioning and Clustering

As warehouses scale, query performance becomes a key concern. It’s not enough to simply store the data—you also need to retrieve it quickly and cost-effectively.

One common optimization is partitioning, which breaks up large tables into smaller, manageable chunks based on a field like date, region, or product category. When a query specifies a filter on that field, the engine can skip over partitions that aren’t relevant, significantly reducing scan times.

Another technique is clustering, which organizes the physical layout of data based on a set of fields that are commonly filtered or joined on. For example, clustering sales records by customer ID can improve performance for queries that retrieve purchase history.

Columnar storage is also key to performance. Unlike row-based storage, which keeps all fields of a record together, columnar formats like those used in BigQuery or Redshift store each column separately. This allows the engine to scan only the columns needed for a query, reducing I/O and speeding up execution.

Data Loading and Refresh Patterns

Getting data into the warehouse is typically done through ETL or ELT processes. These pipelines extract data from source systems, apply transformations, and load the result into warehouse tables.

Loading can happen in batches—say, every hour or once a day—or in micro-batches that simulate near-real-time ingestion. The right frequency depends on your business needs and the capabilities of your orchestration tools.

Incremental loading is often preferred over full reloads. By only processing new or changed records, pipelines reduce load times and warehouse compute costs. This usually requires tracking change data through mechanisms like timestamps or change data capture (CDC).

Warehouse Technologies

Several platforms dominate the modern data warehousing space, each with its strengths.

Snowflake offers a fully managed, multi-cluster architecture with automatic scaling and support for semi-structured data. It separates compute from storage and supports concurrent workloads with minimal tuning.

Google BigQuery is a serverless, query-on-demand platform that excels at ad hoc analytics and scales seamlessly with user demand. It’s ideal for teams that want fast performance without managing infrastructure.

Amazon Redshift provides deep integration with the AWS ecosystem and allows more control over configuration, which can be valuable for teams with specific performance tuning needs.

Each of these platforms supports ANSI SQL, integrates with major BI tools, and offers features for security, monitoring, and data governance.

Wrapping Up

A data warehouse isn’t just a place to store data—it’s the system of record for analytics. Its structure, performance, and accessibility determine how quickly stakeholders can make informed decisions.

Designing and maintaining an effective warehouse requires a thoughtful approach to modeling, data loading, and performance tuning. As your organization grows, so do the expectations placed on your warehouse to handle increasing complexity, scale, and demand for real-time insight.

In the next post, we’ll explore how data lakes differ from warehouses, and how they offer a flexible, scalable foundation for managing large volumes of diverse data types.

Introduction to Data Engineering Concepts | Data Lakes Explained

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

As data volumes grow and the types of data organizations work with become more varied, traditional data warehouses start to show their limits. Structured data fits neatly into tables, but what about videos, logs, images, or JSON documents with unpredictable formats? This is where the concept of a data lake comes into play.

In this post, we’ll explore what a data lake is, how it compares to a data warehouse, and why it’s become a cornerstone of modern data architecture.

What is a Data Lake?

A data lake is a centralized repository designed to store data in its raw form. Whether the data is structured like CSV files, semi-structured like JSON, or unstructured like text or images, the lake accepts it all. It acts as a catch-all layer for every piece of data an organization might want to use for analysis, training models, or historical archiving.

Unlike a data warehouse, which expects a predefined schema and consistent structure, a data lake embraces flexibility. The idea is to collect the data first and figure out how to use it later—a principle often referred to as schema-on-read.

This approach enables data engineers and scientists to access and experiment with data that hasn’t yet been modeled or cleaned. It fosters innovation by removing upfront constraints about how data should look.

Key Characteristics

At its core, a data lake is built on inexpensive, scalable storage—typically object storage like Amazon S3, Azure Data Lake Storage, or Google Cloud Storage. These systems offer the capacity to store petabytes of data without the overhead of traditional database systems.

Because lakes deal with raw data, they don’t enforce strict schemas when data is written. Instead, structure is applied at query time. This allows different teams to interpret the same data in different ways, depending on the analysis they want to perform.

This flexibility is powerful, but it comes with a cost: governance becomes more challenging. Without strong metadata management and data cataloging, lakes can quickly turn into what’s often called a “data swamp”—a cluttered repository that’s hard to navigate or trust.

Data Lakes vs Data Warehouses

The primary difference between data lakes and data warehouses lies in structure and purpose.

Data warehouses are optimized for structured data, curated models, and consistent performance. They serve business users who need reliable access to cleaned, aggregated data for dashboards and reports.

Data lakes are optimized for scale and flexibility. They support raw data, including logs, sensor output, and third-party feeds, making them ideal for machine learning and advanced analytics. While a warehouse is all about predefined questions and structured answers, a lake is about exploration and experimentation.

In practice, many organizations use both. The lake acts as the foundation, storing everything, while the warehouse sits on top as a refined layer for operational analytics. This layered architecture sets the stage for more advanced approaches, such as the data lakehouse, which we'll explore later in this series.

Building and Managing a Data Lake

Creating a data lake involves more than dumping files into storage. A well-functioning lake includes clear organization, access controls, and metadata layers that describe what each dataset is, where it came from, and how it’s used.

Data is often organized into zones. A raw zone stores unprocessed source data. A staging or clean zone contains transformed and validated datasets. A curated zone includes data that’s ready for consumption by analysts or applications.

Maintaining this structure helps manage lifecycle policies, access permissions, and lineage. Cataloging tools like AWS Glue, Apache Hive Metastore, or more modern solutions like Amundsen or DataHub help track what’s in the lake and make it discoverable.

Processing engines like Apache Spark, Presto, or Dremio allow users to query data directly in the lake, using SQL or custom logic. These tools interpret files stored in formats like Parquet, ORC, or Avro, applying structure dynamically based on metadata or inferred schema.

When to Use a Data Lake

A data lake makes the most sense when you’re dealing with large volumes of diverse data types or when you're unsure how the data will be used. It’s particularly valuable in environments focused on research, machine learning, or combining traditional business data with less conventional sources like social media or IoT signals.

However, if you need consistent, curated data for business reporting, a warehouse may be the better choice. Data lakes and warehouses serve different needs, and understanding how they complement each other is key to building a balanced architecture.

In the next post, we’ll look at storage formats and compression—essential building blocks for making data lakes and warehouses efficient, scalable, and cost-effective.

Introduction to Data Engineering Concepts | Storage Formats and Compression

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

When working with large-scale data systems, it's not just what data you store that matters—it's how you store it. The choice of storage format and compression strategy can make a significant difference in performance, cost, and usability. These decisions affect how quickly you can query data, how much storage space you need, and even how compatible your data is with various processing tools.

In this post, we’ll explore the most common data storage formats, the role of compression, and how these choices impact modern data engineering workflows.

Why Storage Format Matters

Raw data often arrives in simple formats like CSV or JSON, and for small volumes, these formats work just fine. But as data grows into gigabytes or terabytes, inefficiencies start to show.

Text-based formats like CSV are easy to read and parse, but they lack schema enforcement, are verbose, and are slow to process in distributed systems. JSON adds some flexibility by allowing nested structures, but it can still be quite large and inefficient when stored at scale.

Columnar formats, by contrast, are designed for analytics. Instead of storing data row by row, they store values column by column. This layout enables faster queries and better compression—especially for workloads that scan only a few columns at a time.

Imagine a table with hundreds of columns, but your query only needs five. With a row-based format, the system must read everything. With a columnar format, it reads just what’s needed. This is a game-changer for performance and cost in systems like data lakes and warehouses.

Common Formats in Practice

Several formats are widely used in data engineering, each with trade-offs.

CSV remains popular due to its simplicity and universal support. But it lacks strong typing and is prone to edge-case issues, such as inconsistent delimiters or quoting problems. It's best used for small datasets or temporary interoperability.

JSON and XML are useful for semi-structured data. JSON, in particular, is common in APIs and logs. However, it’s not space-efficient and can be slow to parse at scale.

Parquet is a columnar format developed by Apache. It's optimized for big data workloads and supports advanced features like nested schemas and predicate pushdown. Parquet is well-supported across tools like Spark, Hive, Dremio, and data warehouses like BigQuery and Snowflake.

Avro is a row-based format with support for schema evolution. It’s often used in streaming applications and data serialization. While it’s not as query-efficient as Parquet, it excels in write-heavy and messaging scenarios.

ORC (Optimized Row Columnar) is similar to Parquet but originally developed for the Hadoop ecosystem. It offers strong compression and performance benefits for read-heavy workloads.

Choosing between these often comes down to the nature of the workload. If you're doing analytics over large datasets, columnar formats like Parquet or ORC are usually the right call. If you're capturing events or streaming messages, Avro might be a better fit.

The Role of Compression

Compression reduces file sizes by encoding repeated or predictable patterns more efficiently. In distributed systems, this saves both storage space and network bandwidth, speeding up data movement and reducing cost.

Compression can be applied at the file level or at the column level (in columnar formats). Modern formats like Parquet support multiple compression codecs, including Snappy, Gzip, Brotli, and Zstd.

Snappy offers fast compression and decompression, making it a good default choice when speed matters more than maximum size reduction. Gzip provides better compression ratios but is slower. Zstd and Brotli strike a balance, offering both speed and compression efficiency.

When choosing a compression strategy, consider the use case. For interactive querying, speed matters, so faster codecs like Snappy are preferred. For archival data or large transfers, stronger compression may save more money in the long run.

Compatibility and Ecosystem Support

Storage format decisions also impact which tools you can use. Most modern data tools support Parquet and Avro natively, but compatibility can vary depending on the processing engine.

For example, if you're building a data lake on S3 and using Apache Spark for processing, Parquet is almost always a safe choice. It integrates well with tools like Hive Metastore, Presto, Trino, and Dremio.

If you’re using Kafka or other message queues, Avro is a common format due to its compactness and schema registry support.

It’s also worth considering schema evolution—how well a format handles changes in the data structure over time. Avro and Parquet both support schema evolution, which allows you to add or remove fields without breaking downstream systems. This is crucial in agile environments where data changes frequently.

Putting It All Together

The best storage strategy balances performance, flexibility, and compatibility. There’s no one-size-fits-all answer, but understanding the characteristics of each format—and how compression affects storage and query speed—allows you to make informed choices.

As data engineers, our job is to pick the right tools for the job, not just default to what’s familiar. Thoughtful decisions at the storage layer can ripple across the entire data stack, affecting cost, speed, and scalability.

In the next post, we’ll turn our attention to data quality and validation—because no matter how well your data is stored, it’s only as good as it is accurate, complete, and trustworthy.

Introduction to Data Engineering Concepts | Data Quality and Validation

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

In any data system, quality is not optional—it’s foundational. No matter how scalable your architecture is, or how fast your queries run, if the underlying data is inaccurate, incomplete, or inconsistent, the results will be misleading. And bad data leads to bad decisions.

This post focuses on data quality and validation. We'll look at what makes data "good," why quality issues emerge, and how engineers can build checks and balances into pipelines to ensure the reliability of their datasets.

Defining Data Quality

At its core, data quality is about trust. Can the data be used confidently for reporting, analytics, or decision-making? While quality is a broad concept, it typically includes several dimensions:

Accuracy: Does the data reflect reality? For example, does a customer record show the correct name and email?
Completeness: Are all required fields populated? Missing data can render entire records useless.
Consistency: Is the data uniform across systems? If two systems say different things about the same event, which one is right?
Timeliness: Is the data fresh enough for its intended purpose? A report showing yesterday’s numbers might be fine—or it might be too late.
Uniqueness: Are there duplicate records that shouldn’t exist?

These attributes form the foundation of what we think of as “high-quality” data. But quality isn't static—it needs to be monitored continuously.

Where Data Quality Breaks Down

Quality issues usually arise at system boundaries. When data moves from one source to another—say, from a transactional database to a warehouse or from an API to a data lake—transformations, encoding issues, and format mismatches can cause subtle errors.

Sometimes data is flawed at the source. A user enters a malformed email address, or a sensor transmits faulty readings due to hardware glitches. Other times, issues emerge downstream, such as when a pipeline fails silently or when schema changes aren’t communicated across teams.

Even well-designed systems can encounter quality problems if the underlying business logic evolves. For example, a rule that defines how revenue is calculated may change, invalidating previous calculations if pipelines aren’t updated accordingly.

The Role of Validation

To combat these issues, validation is key. Validation is the act of checking data against expected rules and assumptions—often before it gets loaded into a final destination.

This can happen at multiple stages of a pipeline. During ingestion, validation might confirm that all required fields are present and formatted correctly. During transformation, it might enforce business rules, such as ensuring that order totals are positive or that timestamps are within reasonable ranges.

Validation can be passive—logging anomalies for review—or active, stopping a pipeline if thresholds are exceeded. Both approaches have their place. In some cases, it's better to allow partial data to flow through and alert the team. In others, it’s critical to block the update to prevent contamination of production datasets.

Tools for Data Quality

Several tools and frameworks have emerged to help engineers define, monitor, and enforce data quality checks.

Great Expectations is one of the most well-known. It allows you to define “expectations” about your data—essentially, assertions about what should be true. These expectations can be validated at runtime, and the results can be logged, visualized, or used to trigger alerts.

Another option is Amazon Deequ, a library built on top of Apache Spark that performs similar validations at scale. It’s particularly useful in large distributed environments where running manual checks would be too costly.

Some orchestration platforms, like Airflow and Dagster, support custom sensors or hooks that let you embed validation logic directly into the DAG. This tight integration makes it easier to halt jobs or notify teams when something goes wrong.

Beyond tools, quality also depends on process. Data contracts, code reviews, and automated testing all contribute to building a culture where quality is prioritized from the start, not added as an afterthought.

Designing for Trust

A key principle in data engineering is that quality doesn't just happen—it must be designed. That means proactively defining what “correct” looks like, instrumenting checks, and making sure failures are surfaced early.

Dashboards and data catalogs can help surface issues. But even more important is visibility: stakeholders need to know when data is delayed, incomplete, or incorrect. Setting up alerts based on data quality metrics helps teams respond quickly before problems reach downstream consumers.

The cost of low-quality data isn't just technical—it's strategic. If users lose faith in the data, they stop relying on it. And once trust is gone, it’s incredibly hard to rebuild.

In the next post, we’ll examine how metadata, lineage, and governance play a role in maintaining data integrity across complex systems. Knowing where your data came from and how it was transformed is just as important as validating its contents.

Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

As data systems grow more complex, understanding where your data came from, how it has changed, and who is responsible for it becomes just as critical as the data itself. It’s not enough to know that a dataset exists—you need to know how it was created, whether it’s trustworthy, and how it fits into the broader system.

In this post, we’ll break down three interconnected concepts—metadata, data lineage, and governance—and explore why they’re essential to building transparent, scalable, and compliant data infrastructure.

What Is Metadata?

Metadata is data about data. It describes the contents, structure, and context of a dataset, giving you the information needed to understand how to work with it.

At the most basic level, metadata includes things like column names, data types, and row counts. But it can go much deeper. Metadata can describe data freshness (when it was last updated), sensitivity (whether it contains personally identifiable information), and ownership (who created or maintains the dataset).

Well-managed metadata serves as a map to your data ecosystem. It helps engineers understand dependencies, enables analysts to find the right datasets, and assists compliance teams in locating sensitive information.

Without metadata, even high-quality data becomes hard to use. Teams end up duplicating effort, making incorrect assumptions, or spending more time asking questions than building insights.

Understanding Data Lineage

Data lineage is the history of how data moves and changes through your systems. It traces the path from the original source—say, a transactional database or API—all the way to its final destination in a dashboard, report, or machine learning model.

Lineage tells you not just where the data is now, but how it got there. Which tables did it pass through? What transformations were applied? Was any filtering, aggregation, or enrichment performed?

This visibility is crucial for several reasons. First, it helps with debugging. When a report shows an unexpected number, lineage lets you trace the logic backwards to find the source of the issue. Second, it supports impact analysis. If a schema changes in a source table, you can immediately see which downstream systems are affected.

In regulated industries, lineage is also a compliance requirement. Auditors often want to see a clear trail from raw data to final output to ensure accuracy, transparency, and accountability.

The Role of Data Governance

Data governance is the set of policies, processes, and roles that ensure data is managed responsibly across an organization. It covers who has access to what data, how it should be handled, and how changes are documented and approved.

Governance is often misunderstood as being purely about control, but it’s really about enabling trust at scale. In small teams, people can rely on informal communication to manage data. In large organizations, clear governance is the only way to prevent chaos.

Good governance defines roles and responsibilities. Who is the data owner? Who approves changes? Who can grant access? It also sets standards for naming, documentation, and data classification so that teams can work together without constant re-alignment.

This becomes even more important in environments with sensitive data. Personally identifiable information (PII), financial records, and health data all come with legal and ethical obligations. Governance ensures these datasets are properly secured, audited, and retained only as long as necessary.

Tools and Practices

To manage metadata, lineage, and governance effectively, many organizations turn to dedicated platforms. Tools like Amundsen, DataHub, and Apache Atlas offer data cataloging and discovery features that make metadata more accessible and actionable.

These platforms often integrate with processing engines and orchestration tools to automatically collect lineage. For example, if a pipeline built in Airflow or dbt modifies a dataset, the lineage graph is updated to reflect that change.

But tools alone aren’t enough. Teams need practices that reinforce good habits—such as documenting changes, defining clear data ownership, and reviewing access permissions regularly.

Automation can help, especially in dynamic environments where datasets are frequently added or updated. But governance must also be embedded into the culture. Engineers, analysts, and stakeholders all play a part in maintaining data integrity and clarity.

Bringing It All Together

Metadata, lineage, and governance are not isolated concerns. Together, they create a foundation for transparency and trust. They help organizations understand what data they have, how it’s being used, and whether it can be relied upon.

Without this foundation, even the best-engineered pipelines can become liabilities. But with it, data becomes a strategic asset—one that teams can build on confidently, securely, and efficiently.

In the next post, we’ll explore how workflow orchestration ties these pieces together, enabling you to manage complex data pipelines reliably across diverse tools and systems.

Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

As data pipelines grow in complexity, managing them manually becomes unsustainable. Whether you're running daily ETL jobs, refreshing dashboards, or processing streaming data in micro-batches, you need a way to coordinate and monitor these tasks reliably. That’s where workflow orchestration comes in.

In this post, we'll explore what orchestration means in the context of data engineering, how it differs from simple job scheduling, and what tools and design patterns help keep data workflows organized, observable, and resilient.

From Scheduling to Orchestration

At the simplest level, scheduling is about running tasks at a certain time. A cron job that triggers a Python script every morning is a form of scheduling. For small pipelines with few dependencies, this can be enough.

But modern data systems rarely involve just one job. Instead, they include chains of tasks—data extractions, file transformations, validation checks, and loads into various targets. These tasks have dependencies, need error handling, and often require conditional logic. This is where orchestration becomes essential.

Workflow orchestration is the discipline of managing task execution across a defined sequence, ensuring that tasks run in the correct order, on time, and with awareness of success or failure. It's not just about launching scripts—it's about understanding how those scripts relate to one another, how they behave under different conditions, and how to recover when something goes wrong.

Directed Acyclic Graphs (DAGs)

Most orchestration systems use the concept of a Directed Acyclic Graph (DAG) to represent workflows. In a DAG, each node represents a task, and edges represent dependencies. The "acyclic" part means there are no loops—each task runs once, and the flow moves in one direction.

This structure allows you to define complex workflows declaratively. For example, you might define a pipeline where data is first extracted from an API, then validated, transformed, and finally loaded into a data warehouse. If any step fails, the system can stop the pipeline, alert the team, or retry the task based on configuration.

DAGs also make it easier to track the status of each component. You can visualize which tasks succeeded, which are still running, and where failures occurred. This visibility is crucial for maintaining trust in your data pipelines.

Common Orchestration Tools

Several orchestration frameworks have become standard in the data engineering ecosystem.

Apache Airflow is one of the most widely adopted tools. It allows users to define DAGs using Python code, which makes it highly flexible and programmable. Airflow includes scheduling, retries, logging, and a web UI for monitoring workflows.

Prefect takes a modern approach by separating the orchestration layer from execution, which makes it more cloud-native and resilient to task failures. Prefect’s focus on observability and developer experience has made it popular for teams managing dynamic workloads.

Dagster emphasizes data assets and type safety. It treats data pipelines as modular, testable units and integrates tightly with modern tooling, including dbt and cloud environments.

Each of these tools supports task dependencies, conditional logic, parallelism, and failure recovery. Choosing the right one often comes down to team preference, operational needs, and ecosystem compatibility.

Best Practices in Workflow Design

Designing orchestration workflows requires more than chaining tasks together. Robust pipelines include thoughtful handling of edge cases and clear observability. That means:

Using retries and timeouts to deal with flaky services or transient failures.
Logging meaningful output so that issues can be diagnosed quickly.
Isolating tasks so that a failure in one part doesn’t compromise unrelated workflows.
Tagging or labeling jobs by function or owner to improve maintainability.

It also means thinking about idempotency. Tasks should be safe to rerun if needed. For example, a data load job that inserts duplicate rows each time it runs will cause problems if retried. Designing tasks to either overwrite cleanly or check for prior completion helps prevent these issues.

Another key practice is modularity. Instead of building large monolithic DAGs, break workflows into reusable components. This makes it easier to test, maintain, and scale your pipelines as your data ecosystem evolves.

Observability and Alerting

A well-orchestrated pipeline doesn’t just run—it tells you how it’s running. Observability is about surfacing the right information at the right time so that engineers can respond to issues quickly.

Good orchestration tools provide dashboards, logs, and metrics. But equally important are alerts that notify the right people when something goes wrong. Alerts should be actionable and avoid noise. A system that sends alerts on every minor warning will eventually be ignored.

Integrating with monitoring platforms like Prometheus, Grafana, or external alerting tools like PagerDuty or Slack helps ensure that teams can respond to problems before they affect end users.

Orchestration as the Backbone

Workflow orchestration isn’t just a technical layer—it’s the backbone of operational data systems. It connects ingestion, transformation, validation, and delivery in a reliable and auditable way. When done well, it turns complex processes into predictable, repeatable workflows that teams can build on confidently.

In the next post, we’ll explore how to build scalable pipelines, including how to think about performance, parallelism, and distribution when dealing with large or fast-growing datasets.

Introduction to Data Engineering Concepts | Building Scalable Pipelines

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

As data volumes increase and workflows grow more interconnected, the ability to build scalable data pipelines becomes essential. It's not enough for a pipeline to work—it needs to keep working as data grows from gigabytes to terabytes, as new sources are added, and as more users rely on the output for decision-making.

In this post, we’ll explore what makes a pipeline scalable, the principles behind designing for growth, and the tools and patterns that data engineers use to manage complexity at scale.

What Do We Mean by Scalability?

Scalability is about more than just performance. It's the ability of a system to maintain its functionality and responsiveness as load increases. In the context of data pipelines, this means handling larger datasets, higher data velocity, and more frequent processing without constant reengineering.

A scalable pipeline gracefully adapts to changes in data size, structure, and frequency. It’s designed in a modular way, so that bottlenecks can be addressed without rewriting the entire system. And it’s observable and maintainable, so issues can be diagnosed before they affect users.

Scalability also involves cost efficiency. Throwing more resources at a slow pipeline might fix the symptoms, but a well-designed system scales intelligently, minimizing unnecessary computation and data movement.

Parallelism and Distribution

One of the core principles behind scalability is parallelism—the ability to split work into independent chunks that can be processed simultaneously.

In batch workflows, this might mean partitioning data by date or region and processing each partition in parallel. In streaming systems, it means dividing incoming data into partitions or shards that are consumed by multiple workers.

Distributed computing frameworks like Apache Spark, Flink, and Dask are designed with this in mind. They break down data into smaller units, distribute them across a cluster of machines, and execute tasks in parallel, tracking dependencies and ensuring consistency across the system.

But parallelism introduces its own challenges. Data skew—when one partition is significantly larger than others—can lead to uneven workloads and poor performance. Effective partitioning strategies and thoughtful job configuration are key to maintaining balance.

Minimizing Data Movement

Another aspect of scalability is reducing how often and how far data moves. Every transfer across a network or system boundary adds latency, cost, and potential failure points.

Where possible, pipelines should process data close to where it's stored. For example, using a query engine like Dremio or Presto to query data directly from object storage avoids the overhead of loading it into a warehouse first.

Materializing only what’s needed, caching intermediate results, and pushing filters down into source systems are all ways to reduce unnecessary computation and movement.

Streaming pipelines, in particular, benefit from minimizing state size and using windowed processing, so that each event is handled quickly and discarded once processed.

Managing Resources

Scalable pipelines require careful resource management. Compute, memory, and I/O all need to be provisioned in a way that meets demand without excessive overhead.

Autoscaling, used in many cloud-native environments, allows processing clusters to grow and shrink based on workload. This is especially valuable for unpredictable or bursty workloads, where fixed infrastructure would either overrun or sit idle.

Monitoring and alerting tools provide visibility into where resources are being used inefficiently. Long-running jobs, slow joins, or excessive data shuffles can all indicate areas where performance tuning is needed.

Tuning batch sizes, controlling concurrency, and using backpressure mechanisms in streaming systems help maintain throughput without overloading infrastructure.

Designing for Change

Scalability isn’t just about today’s workload—it’s about tomorrow’s. Data pipelines should be designed to evolve.

This means avoiding hard-coded assumptions about schema, partitions, or file sizes. It means using configuration over code where possible, and abstracting logic into reusable modules that can be adapted as requirements shift.

Schema evolution support, metadata management, and data contracts between producers and consumers help ensure that changes can be made safely, without breaking downstream systems.

Testing plays a big role here as well. Unit tests for transformations, integration tests for pipeline steps, and data quality checks all contribute to a system that can grow without becoming brittle.

Bringing It All Together

Scalable pipelines don’t happen by accident. They’re the result of intentional design choices that account for volume, velocity, and variability.

By embracing parallelism, minimizing data movement, managing resources effectively, and planning for change, data engineers can build pipelines that not only meet today’s demands but are ready for tomorrow’s challenges.

In the next post, we’ll look at how DevOps principles apply to data engineering—covering CI/CD, infrastructure as code, and the tools that support reliable and automated data deployments.

Introduction to Data Engineering Concepts | DevOps for Data Engineering

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

As data systems grow more complex and interconnected, the principles of DevOps—long applied to software engineering—have become increasingly relevant to data engineering. Continuous integration, infrastructure as code, testing, and automation aren’t just for deploying apps anymore. They’re essential for delivering reliable, maintainable, and scalable data pipelines.

In this post, we’ll explore how DevOps practices translate into the world of data engineering, why they matter, and what tools and techniques help bring them to life in modern data teams.

Bridging the Gap Between Code and Data

At the heart of DevOps is the idea that development and operations should be integrated. In traditional software development, this means automating the steps from writing code to running it in production. For data engineering, the challenge is similar—but the output isn't always a user-facing app. Instead, it's pipelines, transformations, and datasets that power reports, dashboards, and machine learning models.

The core question becomes: how do we ensure that changes to data workflows are tested, deployed, and monitored with the same rigor as application code?

The answer lies in adopting DevOps-inspired practices like version control, automated testing, continuous deployment, and infrastructure automation—all tailored to the specifics of data systems.

Version Control for Pipelines and Configurations

Just like in software engineering, all code that defines your data infrastructure—SQL queries, transformation logic, orchestration DAGs, and even schema definitions—should live in version-controlled repositories.

This makes it easier to collaborate, review changes, and roll back when something breaks. Tools like Git, combined with platforms like GitHub or GitLab, provide the foundation. Branching strategies and pull requests help teams manage change in a structured, auditable way.

Even configurations—such as data source definitions or schedule timings—can and should be versioned, ideally alongside the pipeline logic they support.

Continuous Integration and Testing

Data pipelines are code, and they should be tested like code. This includes unit tests for transformation logic, integration tests for full pipeline runs, and data quality checks that assert assumptions about the shape and content of your data.

CI pipelines, powered by tools like GitHub Actions, GitLab CI, or Jenkins, can run these tests automatically on each commit or pull request. They ensure that changes don’t break existing functionality or introduce regressions.

Testing data workflows is more nuanced than testing application logic. It often involves staging environments with synthetic or sample data, mocking external dependencies, and verifying outputs across time windows. But the goal is the same: catch problems early, not after they hit production.

Infrastructure as Code

Managing infrastructure manually—whether it’s a Spark cluster, an Airflow deployment, or a cloud storage bucket—doesn’t scale. Infrastructure as code (IaC) provides a way to define your environment in declarative files that can be versioned, reviewed, and deployed automatically.

Tools like Terraform, Pulumi, and CloudFormation allow data teams to define compute resources, networking, permissions, and even pipeline configurations as code. Combined with CI/CD, IaC enables repeatable deployments, easier disaster recovery, and consistent environments across dev, staging, and production.

IaC also helps in tracking infrastructure changes over time. When something breaks, you can look at the exact commit that introduced the change—not just guess what might have gone wrong.

Continuous Deployment for Pipelines

Once code is tested and approved, it needs to be deployed. Continuous deployment automates this step, pushing new pipeline definitions or transformation logic into production systems with minimal manual intervention.

In practice, this might mean updating DAGs in Airflow, deploying dbt models, or rolling out new configurations to a Kafka stream processor. The process should include validation steps, such as verifying schema compatibility or testing data output in a sandbox environment before it goes live.

Feature flags and gradual rollouts—techniques borrowed from application development—can also be applied to data. They allow teams to test changes on a subset of data or users before promoting them system-wide.

Monitoring and Incident Response

Finally, DevOps emphasizes the importance of monitoring and observability. Data pipelines need the same treatment. Logs, metrics, and alerts should provide insight into pipeline health, performance, and failures.

Tools like Prometheus, Grafana, and cloud-native observability platforms can be integrated with orchestration tools to expose runtime metrics. Custom dashboards can show pipeline durations, success rates, and error counts. Alerts can notify teams when jobs fail or when output data violates expectations.

Just as importantly, incidents should feed back into improvement. Postmortems, runbooks, and blameless retrospectives help teams learn from failures and evolve their systems.

Shifting the Culture

Adopting DevOps for data engineering is as much about culture as it is about tools. It means treating data workflows with the same discipline as software systems—building, testing, deploying, and monitoring them in automated, repeatable ways.

This cultural shift leads to faster iterations, fewer outages, and more confidence in the data products that teams rely on. It also reduces the operational load on engineers, freeing them to focus on value creation instead of firefighting.

In the next post, we’ll step back and look at the cloud ecosystem that underpins much of this work. Understanding the role of managed services and cloud-native tools is key to building a modern, agile data platform.

Introduction to Data Engineering Concepts | Cloud Data Platforms and the Modern Stack

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

The cloud has transformed how organizations approach data engineering. What once required physical servers, manual provisioning, and heavyweight infrastructure can now be spun up in minutes with managed, scalable services. But with this convenience comes complexity—deciding how to compose the right mix of tools and platforms for your data workflows.

In this post, we’ll explore what defines the modern data stack, how cloud platforms like AWS, GCP, and Azure fit into the picture, and what principles guide the design of flexible, cloud-native data architectures.

Moving Beyond On-Premise

In traditional, on-premise data systems, teams had to manage everything themselves—hardware, networking, databases, storage, and backups. Scaling required buying more servers. Upgrades were slow, and experimentation was costly.

Cloud platforms shifted this model. Infrastructure became elastic. Managed services replaced self-hosted databases and batch processing engines. What used to take weeks could now be done in hours. This shift enabled data engineers to focus more on business logic and less on infrastructure maintenance.

But while the cloud solved many problems, it also introduced new decisions. With so many tools available, how do you choose the right combination? That’s where the concept of the modern data stack comes in.

What Is the Modern Data Stack?

The modern data stack refers to a collection of tools—often cloud-native—that work together to support the full data lifecycle: ingestion, transformation, storage, orchestration, and analysis.

Typically, this stack includes:

A cloud data warehouse like Snowflake, BigQuery, or Redshift
An ingestion tool such as Fivetran, Airbyte, or custom streaming connectors
A transformation framework like dbt
An orchestration platform like Airflow or Prefect
BI tools such as Looker, Mode, or Tableau

These tools are designed to be modular and API-driven. You can swap components as your needs evolve, without having to rebuild the entire system. They also tend to embrace SQL, making them accessible to a broader range of users, including analysts and analytics engineers.

This composability is powerful, but it requires thoughtful integration. Data engineers must understand how data flows across services, how metadata is preserved, and where bottlenecks can emerge.

Managed Services in the Cloud

Each major cloud provider offers a suite of services tailored to data engineering.

On AWS, services like S3 (storage), Glue (ETL), Redshift (warehousing), and Kinesis (streaming) form the core building blocks. AWS is known for its breadth and flexibility, making it a strong choice for teams that want control and are comfortable managing complexity.

Google Cloud Platform (GCP) centers around BigQuery, a serverless, high-performance data warehouse. Paired with Dataflow (streaming and batch processing), Pub/Sub (messaging), and Looker (BI), GCP offers a tight integration between services with a focus on simplicity and scale.

Microsoft Azure provides tools like Synapse Analytics, Data Factory, and Event Hubs. It often appeals to enterprise environments already invested in Microsoft’s ecosystem, offering deep integration with Active Directory, Power BI, and other services.

Each platform brings its own pricing models, performance characteristics, and operational trade-offs. Choosing one often comes down to organizational context—existing infrastructure, skillsets, and vendor relationships.

Designing for Agility

A key advantage of the cloud is its ability to support experimentation. You can test new tools, build proof-of-concepts, and iterate quickly without long procurement cycles or sunk infrastructure costs.

This agility enables teams to build for today while planning for tomorrow. For example, a team might start with batch ingestion and transformation using dbt and Airflow. As data needs grow, they can add streaming layers with Kafka and Spark, or move toward a lakehouse architecture using Iceberg and Dremio.

To design for agility, it’s important to decouple systems where possible. Avoid hard-wiring logic across tools. Use metadata and configuration layers to manage pipeline logic. Embrace standards like Parquet or Arrow to ensure interoperability between tools.

Observability and governance also become more important in a distributed cloud environment. Knowing where your data is, how it’s being used, and who has access requires integrated monitoring, logging, and metadata management.

The Cloud is Not Just a Hosting Model

Adopting cloud data platforms is not just about moving infrastructure off-premise—it’s about rethinking how teams operate. Cloud-native architectures prioritize scalability, flexibility, and automation.

They allow you to treat data as a product, with well-defined interfaces, quality guarantees, and ownership. They enable collaboration across roles—engineers, analysts, and scientists—by providing shared platforms and standardized workflows.

Ultimately, the modern data stack is not a fixed set of tools, but a mindset. It's about building systems that are composable, observable, and adaptable. It’s about enabling fast iteration without sacrificing reliability.

In the next post, we’ll shift into the final phase of this series and explore the evolution toward data lakehouse architectures—what they are, why they matter, and how they unify the best of both lakes and warehouses.

Introduction to Data Engineering Concepts | Data Lakehouse Architecture Explained

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

Data lakes and data warehouses each brought strengths and limitations to the way organizations manage analytics. Lakes offered flexibility and scale, but lacked consistency and performance. Warehouses delivered speed and structure, but often at the cost of rigidity and duplication. The data lakehouse aims to unify the best of both worlds.

In this post, we’ll explore what a data lakehouse is, how it differs from its predecessors, and why it represents a fundamental shift in modern data architecture.

The Problem with Separate Systems

Historically, data teams maintained two separate systems: a data lake for raw, large-scale data and a warehouse for clean, curated analytics. This split introduced a number of challenges.

Data had to be copied and transformed between systems. Pipelines became complex and brittle, often requiring multiple processing steps to move data from lake storage into a format usable by the warehouse. Governance and metadata management were fragmented. And teams ended up managing duplicate logic in two places, increasing both cost and risk.

This led to a common problem: organizations had access to a lot of data, but not in a way that was fully consistent, trustworthy, or timely.

What is a Lakehouse?

A lakehouse is a single data architecture that combines the scalability and cost-efficiency of a data lake with the data management features of a warehouse. Instead of maintaining separate systems for raw and curated data, a lakehouse enables you to store all data in one place—typically an object store like S3 or ADLS—while layering in transactional guarantees, schema enforcement, and performance optimizations.

The core idea is to treat the lake as the foundation, and then build capabilities on top that make it feel like a warehouse: support for SQL queries, fine-grained access controls, data versioning, and support for BI tools.

With a lakehouse, you can ingest raw data, apply transformations, and serve both data scientists and business analysts from the same platform—without having to move or duplicate data between systems.

Key Capabilities

A few innovations make the lakehouse model possible:

First, table formats like Apache Iceberg and Delta Lake introduce ACID transactions to files stored in data lakes. This means you can safely update, insert, and delete records with consistency, even across distributed systems.

Second, query engines like Dremio, Trino, and Starburst have matured to the point where they can run fast, complex SQL queries directly against files in the lake—especially when using efficient columnar formats like Parquet.

Third, metadata and cataloging layers have improved, enabling better schema management, lineage tracking, and discovery across lakehouse tables.

Together, these advancements bridge the gap between raw storage and structured analytics, making it possible to build a cohesive data platform without compromise.

Benefits of the Lakehouse Approach

One of the most compelling benefits of a lakehouse is simplification. Instead of building multiple pipelines to synchronize data between systems, teams can work from a single source of truth. This reduces latency, lowers operational complexity, and improves data consistency.

Lakehouses are also cost-effective. Object storage is cheaper and more scalable than traditional databases. And by avoiding the need to load data into separate warehouses, you eliminate redundant storage and computation.

From a flexibility standpoint, the lakehouse supports a wide range of use cases—from batch analytics to interactive SQL to machine learning—all from the same underlying data.

Importantly, the lakehouse model supports open standards. With formats like Iceberg, you’re not locked into a single vendor’s ecosystem. Your data remains portable, and you can build your stack using best-of-breed components.

A New Foundation for the Future

The data lakehouse is more than a marketing term—it represents a practical response to the needs of modern data teams. As data volumes continue to grow, and as organizations seek faster, more reliable insights, the need for unified, scalable architectures becomes clear.

By combining the raw power of data lakes with the structure and performance of data warehouses, the lakehouse offers a way to do more with less—less duplication, less movement, and less friction.

In the next post, we’ll dig deeper into the technologies that make the lakehouse possible, starting with Apache Iceberg, Apache Arrow, and Apache Polaris. These tools form the foundation of many modern analytic platforms and help bring the lakehouse vision to life.

Introduction to Data Engineering Concepts | Apache Iceberg, Arrow, and Polaris

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

As the data lakehouse ecosystem matures, new technologies are emerging to close the gap between raw, scalable storage and the structured, governed world of traditional analytics. Apache Iceberg, Apache Arrow, and Apache Polaris are three such technologies—each playing a distinct role in enabling high-performance, cloud-native data platforms that prioritize openness, flexibility, and consistency.

In this post, we’ll explore what each of these technologies brings to the table and how they work together to power modern data workflows.

Apache Iceberg: The Table Format That Changes Everything

Apache Iceberg is more than just a file format—it’s a table format designed to bring SQL-like features to cloud object storage. In traditional data lakes, data is stored in files, but there’s no built-in concept of a table. This makes operations like updates, deletes, or time travel difficult to implement consistently.

Iceberg solves that by introducing a transactional metadata layer. Tables are made up of snapshots, each pointing to a set of manifest files that describe the underlying data files. Every time data is written or updated, a new snapshot is created, and the metadata is atomically updated.

This architecture enables reliable schema evolution, partition pruning, and time travel. It also supports concurrent writes across engines, making Iceberg a foundational layer for scalable, multi-engine data platforms.

Importantly, Iceberg is engine-agnostic. Spark, Flink, Trino, Snowflake, and Dremio all support reading and writing to Iceberg tables, which allows data teams to avoid vendor lock-in and build modular systems.

Apache Arrow: A Universal Memory Format

If Iceberg handles data at rest, Apache Arrow handles data in motion. Arrow is a columnar in-memory format optimized for analytical processing. It allows systems to share data across process boundaries without serialization overhead, which dramatically reduces latency in data transfers.

In practice, Arrow powers faster execution of queries, especially in environments where performance is critical. Engines like Dremio and frameworks like pandas or Apache Flight use Arrow to move data between components efficiently.

Because Arrow defines a common representation for tabular data in memory, it allows tools built in different languages and frameworks to interoperate seamlessly. That’s a big deal in heterogeneous environments where Python, Java, and C++ may all play a role in the same workflow.

Together, Iceberg and Arrow represent a powerful separation of concerns: Arrow optimizes processing in RAM, while Iceberg provides the transactional storage layer on disk.

Apache Polaris: The Missing Catalog Layer

As Iceberg adoption grows, managing Iceberg tables across distributed query engines becomes a challenge. That’s where Apache Polaris comes in.

Polaris is an implementation of the Apache Iceberg REST catalog specification. It provides a centralized service for managing metadata about Iceberg tables and their organizational structure. Instead of having every engine implement its own catalog logic, Polaris provides a shared layer that orchestrates access across tools like Spark, Flink, Trino, and Snowflake.

At the heart of Polaris is the concept of a catalog—a logical container for Iceberg tables, configured to point to your cloud storage. Polaris supports both internal and external catalogs. Internal catalogs are fully managed within Polaris, while external catalogs sync with systems like Snowflake or Dremio Arctic. This flexibility lets you bring your existing Iceberg assets under centralized governance without locking them in.

Polaris organizes tables into namespaces, which are essentially folders within a catalog. These namespaces can be nested to reflect organizational or project hierarchies. Within a namespace, you register Iceberg tables, which can then be accessed by multiple engines through a consistent API.

To connect to Polaris, engines use service principals—authenticated entities with specific privileges. These principals are grouped into principal roles, which receive access rights from catalog roles. This role-based access control (RBAC) system allows for fine-grained security across catalogs, namespaces, and tables.

What makes Polaris especially powerful is its ability to vend temporary credentials during query execution. When a query runs, Polaris provides secure access to the underlying storage without exposing long-term cloud credentials. This mechanism, known as credential vending, ensures both security and operational flexibility.

A Unified Ecosystem

Together, Apache Iceberg, Arrow, and Polaris create a cohesive environment where data can be stored, processed, and accessed consistently and securely—regardless of the engine being used.

Iceberg brings data warehouse-like capabilities to cloud storage. Arrow enables high-performance, memory-efficient processing across languages and systems. Polaris acts as the control plane, coordinating access and governance.

This architecture aligns with the ideals of the data lakehouse: open standards, decoupled compute and storage, and interoperability across tools. By building on these technologies, organizations can future-proof their data platforms while empowering teams to work with the tools they prefer.

In the next and final post in this series, we’ll look at Dremio—a platform that ties these components together to deliver interactive, self-service analytics directly on the data lake, without moving data or duplicating logic.

Introduction to Data Engineering Concepts | The Power of Dremio in the Modern Lakehouse

Alex Merced — Fri, 02 May 2025 09:00:00 GMT

Free Resources

As organizations shift toward data lakehouse architectures, the question isn’t just how to store massive volumes of data—it’s how to optimize it for fast, reliable access without adding complexity or operational overhead. Dremio addresses this challenge head-on by combining performance, governance, and openness into a platform built natively on Apache Iceberg, Apache Arrow, and Apache Polaris.

In this final post of our series, we’ll explore how Dremio ties together the technologies we've discussed—like clustering, reflections, and cataloging—into an integrated solution for modern data engineering. We’ll cover what makes Dremio unique, how its latest innovations like Iceberg Clustering and Autonomous Reflections work, and why these capabilities are a breakthrough for data teams aiming to do more with less.

Built for the Modern Stack

Dremio isn't just a SQL engine—it’s a full data platform built for the lakehouse era. It operates directly on data stored in open formats like Parquet and Iceberg, using Apache Arrow for in-memory performance and Apache Polaris for metadata management and governance. The result is a platform that offers sub-second queries, native support for open standards, and a unified experience across ingestion, transformation, exploration, and security.

Instead of requiring teams to move data into a proprietary warehouse, Dremio enables query federation across lakes, catalogs, and traditional databases. Whether your data lives in S3, GCS, Azure, or multiple warehouses, Dremio can connect, query, and govern it—all without duplication or data movement.

But what truly sets Dremio apart is its focus on intelligent automation and data layout optimization. Let’s break down how these features work.

Iceberg Clustering: Smarter Data Organization

As datasets grow, traditional partitioning strategies fall short. Over-partitioning leads to a flood of small files. Under-partitioning causes massive scan overhead. Dremio introduces Iceberg Clustering to address this gap.

Instead of dividing data into rigid partitions, clustering organizes rows based on column value proximity using Z-ordering, a type of space-filling curve. This technique braids together bits from multiple columns to form an index that preserves locality. The closer the index values, the closer the original rows were in value space—making it easier for the engine to skip irrelevant data.

By clustering non-partitioned tables, Dremio can dramatically reduce the number of data files and row groups scanned during queries. The result: faster performance without the rigidity or complexity of traditional partitioning.

This process is incremental and adaptive. Dremio monitors data file overlap (measured via clustering depth) and selectively rewrites files to restore efficient layout. You don’t have to re-cluster everything or worry about perfect partition granularity—Dremio handles it dynamically and intelligently.

Autonomous Reflections: AI for Query Optimization

Materialized views are great—until you have to decide which ones to create, maintain, and drop. Dremio automates this process with Autonomous Reflections, which monitor your workloads, identify performance bottlenecks, and generate pre-aggregated or pre-filtered views to accelerate queries.

The system analyzes usage patterns and query plans, scores potential reflections based on estimated time savings, and creates only those that deliver meaningful impact. It even keeps them up to date using live metadata refresh and incremental updates, ensuring performance gains without sacrificing freshness.

Reflections are created, scored, and dropped automatically based on cost-benefit analysis, with strict guardrails to avoid wasting resources. This isn’t just automation—it’s intelligent, usage-aware optimization.

With Dremio’s Autonomous Reflections, query acceleration becomes invisible to the user. Queries run faster, dashboards load quicker, and teams no longer need to guess which workloads justify a materialized view. The platform adapts as your usage changes.

Governance and Discoverability with Polaris

Managing Iceberg tables at scale requires more than just metadata tracking—it requires unified governance. Dremio’s integration with Apache Polaris gives teams a central catalog that enforces access controls, tracks lineage, and supports multi-engine access through open REST protocols.

Whether you’re using Spark, Trino, Flink, or Dremio itself, Polaris provides a consistent layer for managing catalogs, namespaces, and Iceberg tables. Service principals and RBAC ensure secure access, while credential vending allows query engines to read data without exposing cloud credentials.

By offering a unified metastore for all your Iceberg assets, Polaris makes it easier to scale governance and integrate with diverse compute engines, all while maintaining data sovereignty and visibility.

AI-Ready Data, Out of the Box

As data volumes soar and AI workloads increase, organizations need data platforms that deliver speed and clarity—not maintenance overhead. Dremio’s new features don’t just optimize query performance; they also support AI and analytics with intelligent automation, semantic search, and unified metadata.

AI-Enabled Semantic Search lets users discover datasets using plain language, not SQL. This reduces time spent hunting for data and accelerates exploration for analysts and data scientists alike. Combined with reflections and clustering, the platform ensures these queries return results fast.

And because Dremio is built on open standards—Iceberg, Arrow, and Polaris—you can trust that your data architecture will remain portable, interoperable, and vendor-neutral.

Real-World Results

Dremio has already demonstrated the power of this approach internally. After deploying clustering and autonomous reflections across its own internal lakehouse, Dremio saw:

80% of dashboards accelerated automatically
10x reduction in 90th percentile query times
30x improvement in CPU efficiency per query
Substantial infrastructure savings by right-sizing compute resources

These improvements weren’t the result of hand-tuning or custom engineering. They were achieved through intelligent automation—something every team can now access.

Conclusion

Data lakehouses offer unmatched flexibility, but performance and manageability have long remained pain points. With features like Iceberg Clustering, Autonomous Reflections, and Polaris Catalog, Dremio turns the lakehouse into a high-performance, governed, and self-optimizing platform.

For data engineers, this means fewer manual interventions, faster time-to-insight, and greater confidence in how data is delivered. For analysts and AI teams, it means sub-second queries and easy access to the data they need—no pipeline delays, no tuning required.

As the final stop in this series, Dremio represents the culmination of modern data engineering principles: openness, automation, and efficiency. If you're building on Iceberg and want to unlock its full potential, Dremio offers a platform designed not just to support your architecture, but to elevate it.

To see it in action, try Dremio for free or explore the latest launch to learn how these capabilities can help your team build a faster, smarter lakehouse.

A Journey from AI to LLMs and MCP - 10 - Sampling and Prompts in MCP — Making Agent Workflows Smarter and Safer

Alex Merced — Mon, 14 Apr 2025 09:00:00 GMT

Free Resources

We’ve now seen how the Model Context Protocol (MCP) allows LLMs to read resources and call tools—giving them access to both data and action.

But what if your MCP server needs the LLM to make a decision?

What if it needs to:

Analyze a file before running a tool?
Draft a message for approval?
Ask the model to choose between options?

That’s where Sampling comes in.

And what if you want to give the user—or the LLM—reusable, structured prompt templates for common workflows?

That’s where Prompts come in.

In this final post of the series, we’ll explore:

How sampling allows servers to request completions from LLMs
How prompts enable reusable, guided AI interactions
Best practices for both features
Real-world use cases that combine everything we’ve covered so far

What Is Sampling in MCP?

Sampling is the ability for an MCP server to ask the host to run an LLM completion—on behalf of a tool, prompt, or workflow.

It lets your server say:

“Hey, LLM, here’s a prompt and some context. Please respond.”

Why is this useful?

You can generate intermediate reasoning steps
Let the model propose actions before executing them
Create more natural multi-turn agent workflows
Maintain human-in-the-loop approval and visibility

Sampling Flow

Here’s the typical lifecycle:

The server sends a sampling/createMessage request
The host (Claude Desktop, etc.) can review or modify the prompt
The host runs the LLM completion
The result is sent back to the server

This architecture puts control and visibility in the hands of the user, even when the agent logic runs server-side.

✉️ Message Format

Here’s an example sampling/createMessage request:

{
  "messages": [
    {
      "role": "user",
      "content": {
        "type": "text",
        "text": "Please summarize this log file."
      }
    }
  ],
  "systemPrompt": "You are a helpful developer assistant.",
  "includeContext": "thisServer",
  "maxTokens": 300
}

The host chooses which model to use, what context to include, and whether to show the prompt to the user for confirmation.

Response:

{
  "model": "claude-3-sonnet",
  "role": "assistant",
  "content": {
    "type": "text",
    "text": "The log file contains several timeout errors and warnings related to database connections."
  }
}

Now the server can act on that response—log it, return it as tool output, or chain it into another step.

Best Practices for Sampling

Best Practice Why It Matters

Use clear system prompts Guides model behavior contextually
Limit tokens Prevent runaway completions
Structure responses Enables downstream parsing (e.g. JSON, bullets)
Include only relevant context Keep prompts focused and cost-effective
Respect user control The host mediates the actual LLM call

What Are Prompts in MCP?

Prompts are reusable, structured templates that servers can expose to clients.

Think of them like slash commands or predefined workflows:

Pre-filled with helpful defaults
Accept arguments (e.g. "project name", "file path")
Optionally include embedded resources
Surface in the client UI

Prompts help users and LLMs collaborate efficiently by standardizing useful tasks.

✨ Prompt Structure

Prompts have:

A name (identifier)
A description (for discovery)
A list of arguments (optional)
A template for generating messages

Example:

{
  "name": "explain-code",
  "description": "Explain how this code works",
  "arguments": [
    {
      "name": "language",
      "description": "Programming language",
      "required": true
    },
    {
      "name": "code",
      "description": "The code to analyze",
      "required": true
    }
  ]
}

Clients use:

prompts/list to discover prompts
prompts/get to resolve a prompt and arguments into messages

Dynamic Prompt Example

A server might expose:

{
  "name": "analyze-logs",
  "description": "Summarize recent logs and detect anomalies",
  "arguments": [
    {
      "name": "timeframe",
      "required": true
    }
  ]
}

When the user (or LLM) runs it with:

{
  "timeframe": "1h"
}

The resolved prompt could include:

A message like: “Please summarize the following logs from the past hour.”
An embedded resource (e.g. logs://recent?timeframe=1h)
Output ready for sampling

Sampling + Prompts = Dynamic Workflows

When you combine prompts + sampling + tools, you unlock real agent behavior.

Example Workflow:

User selects prompt: "Analyze logs and suggest next steps"
Server resolves the prompt and calls sampling/createMessage
LLM returns: “The logs show repeated auth failures. Suggest checking OAuth config.”
Server calls tools/call to run check_auth_config
LLM reviews the result and writes a summary

All controlled via:

Standardized MCP messages
User-visible approvals
Modular server logic

🔐 Security and Control

Feature	How It's Handled
Prompt visibility	Clients decide which prompts to expose
Sampling review	Hosts can show/reject sampling requests
Input validation	Servers validate prompt arguments
Model usage control	Hosts select models and limit token costs
Prompt injection risks	Validate user inputs, escape content if needed

🧠 Why These Matter for AI Agents

Capability	Sampling Provides	Prompts Provide
Decision-making	Dynamic LLM completions	Guided, structured input
Flexibility	Server can request help anytime	Users can run reusable workflows
Interactivity	Chain actions with feedback	Improve LLM collaboration
Composability	Mix prompts + tools + resources	Enable custom interfaces

🧩 Wrapping It All Together

Over this 10-part series, we’ve explored the full landscape of AI agent development using MCP:

✅ LLMs and how they work
✅ Fine-tuning, prompting, and RAG
✅ Agent frameworks and limitations
✅ MCP’s architecture and interoperability
✅ Resources and tools
✅ Prompts and sampling

MCP gives us standardized, modular building blocks for creating AI agents that are:

Portable across environments
Decoupled from model providers
Secure, observable, and controlled

A Journey from AI to LLMs and MCP - 9 - Tools in MCP — Giving LLMs the Power to Act

Alex Merced — Sun, 13 Apr 2025 09:00:00 GMT

Free Resources

In the previous post, we looked at Resources in the Model Context Protocol (MCP): how LLMs can securely access real-world data to ground their understanding. But sometimes, reading isn’t enough.

Sometimes, you want the model to do something.

That’s where Tools in MCP come in.

In this post, we’ll explore:

What tools are in MCP
How tools are discovered and invoked
How LLMs can use tools (with user control)
Common tool patterns and security practices
Real-world examples: from file system commands to API wrappers

Let’s dive in.

What Are Tools in MCP?

Tools are executable functions that an LLM (or the user) can call via the MCP client. Unlike resources—which are passive data—tools are active operations.

Examples include:

Running a shell command
Calling a REST API
Summarizing a document
Posting a GitHub issue
Triggering a build process

Each tool includes:

A name (unique identifier)
A description (for UI/model understanding)
An input schema (JSON schema describing expected parameters)

Tools allow models to interact with the world beyond natural language—under user oversight.

Discovering Tools

Clients can list available tools via: tools/list

Example response:

{
  "tools": [
    {
      "name": "calculate_sum",
      "description": "Add two numbers together",
      "inputSchema": {
        "type": "object",
        "properties": {
          "a": { "type": "number" },
          "b": { "type": "number" }
        },
        "required": ["a", "b"]
      }
    }
  ]
}

This allows clients (and LLMs) to decide which tools are available and how to call them properly.

⚙️ Calling a Tool

To execute a tool, the client sends:

tools/call

With this payload:

{
  "name": "calculate_sum",
  "arguments": {
    "a": 3,
    "b": 5
  }
}

The server responds with:

{
  "content": [
    {
      "type": "text",
      "text": "8"
    }
  ]
}

That’s it! The LLM can now use this output in a multi-step reasoning chain.

Model-Controlled Tool Use

Tools are designed to be invoked by models automatically. The host mediates this interaction with:

Approval flows (user-in-the-loop)
Permission gating
Logging and auditing

This is what enables “agentic behavior.” For example:

Claude sees a CSV file and decides to call analyze_csv to compute averages—without a user explicitly requesting it.

Tool Design Patterns

Let’s look at some common and powerful tool types:

System Tools

{
  "name": "run_command",
  "description": "Execute a shell command",
  "inputSchema": {
    "type": "object",
    "properties": {
      "command": { "type": "string" },
      "args": {
        "type": "array",
        "items": { "type": "string" }
      }
    }
  }
}

Use case: Let the LLM grep a log file, or check system uptime.

API Integrations

{
  "name": "create_github_issue",
  "description": "Open a new issue on GitHub",
  "inputSchema": {
    "type": "object",
    "properties": {
      "repo": { "type": "string" },
      "title": { "type": "string" },
      "body": { "type": "string" }
    }
  }
}

Use case: Let an AI dev assistant file bugs or suggest changes.

Data Analysis

{
  "name": "summarize_csv",
  "description": "Summarize a CSV file",
  "inputSchema": {
    "type": "object",
    "properties": {
      "filepath": { "type": "string" }
    }
  }
}

Use case: Let the LLM analyze performance metrics or user data.

Security Best Practices

Giving LLMs the ability to take action means security is critical. Here’s how to stay safe:

Validate all input Use detailed JSON schemas

Sanitize input (e.g., file paths, commands)

Use access controls Gate sensitive tools behind roles

Allow user opt-in or approval

Log and monitor usage Track which tools are used, with what arguments

Log errors and output for audit trails

Handle errors gracefully Return structured errors inside the result, not just raw exceptions. This helps the LLM adapt.

{
  "isError": true,
  "content": [
    {
      "type": "text",
      "text": "Error: File not found."
    }
  ]
}

Example: Implementing a Tool Server in Python

@mcp.tool()
async def get_weather(city: str) -> str:
    """Return current weather for a city."""
    data = await fetch_weather(city)
    return f"The temperature in {city} is {data['temp']}°C."

This tool will automatically appear in the tools/list response and can be invoked by the LLM or user.

Why Tools Matter for Agents

Agents aren’t just chatbots—they're interactive systems. Tools give them the ability to:

Take real-world actions
Build dynamic workflows
Chain reasoning across multiple steps
Drive automation in safe, auditable ways

Combined with resources, prompts, and sampling, tools make LLMs feel like collaborative assistants, not just text predictors.

Recap: Tools in MCP

Concept Description
Tool definition Name, description, and input schema
Invocation tools/call with arguments
Output Text or structured response
Use case examples Shell commands, API calls, code generation, analysis
Security guidelines Validate input, log usage, gate sensitive actions

Coming Up Next: Sampling and Prompts — Letting the Server Ask the Model for Help

In the final two posts of this series, we’ll explore:

✅ Sampling — How servers can request completions from the LLM during workflows ✅ Prompts — Reusable templates for user-driven or model-driven actions

Tools give LLMs the power to act. With proper controls and schemas, they become safe, composable building blocks for real-world automation.

A Journey from AI to LLMs and MCP - 8 - Resources in MCP — Serving Relevant Data Securely to LLMs

Alex Merced — Sat, 12 Apr 2025 09:00:00 GMT

Free Resources

In the previous post, we explored the architecture of the Model Context Protocol (MCP)—a flexible, standardized way to connect LLMs to tools, data, and workflows. One of MCP’s most powerful capabilities is its ability to expose resources to language models in a structured, secure, and controllable way.

In this post, we’ll dive into:

What MCP resources are
How they’re discovered and accessed
Text vs binary resources
Dynamic templates and subscriptions
Best practices for implementation and security

If you want to give LLMs real, relevant context from your systems—without compromising safety or control—resources are the foundation.

What Are Resources in MCP?

Resources represent data that a model or client can read.

This might include:

Local files (e.g. file:///logs/server.log)
Database records (e.g. postgres://db/customers)
Web content (e.g. https://api.example.com/data)
Images or screenshots (e.g. screen://localhost/monitor1)
Structured system data (e.g. logs, metrics, config files)

Each resource is identified by a URI, and can be read, discovered, and optionally subscribed to for updates.

Resource Discovery

Clients can ask a server to list available resources using: resources/list

The server responds with an array of structured metadata:

{
  "resources": [
    {
      "uri": "file:///logs/app.log",
      "name": "Application Logs",
      "description": "Recent server logs",
      "mimeType": "text/plain"
    }
  ]
}

Clients (or users) can browse these like a menu, selecting what context to send to the model.

Resource Templates

In addition to static lists, servers can expose URI templates using RFC 6570 syntax:

{
  "uriTemplate": "file:///logs/{date}.log",
  "name": "Log by Date",
  "description": "Access logs by date (e.g., 2024-04-01)",
  "mimeType": "text/plain"
}

This allows dynamic access to parameterized content—great for APIs, time-based logs, or file hierarchies.

Reading a Resource

To retrieve the content of a resource, clients use:

resources/read With a payload like:

{
  "uri": "file:///logs/app.log"
}

The server responds with the content in one of two formats:

Text Resource

{
  "contents": [
    {
      "uri": "file:///logs/app.log",
      "mimeType": "text/plain",
      "text": "Error: Timeout on request...\n"
    }
  ]
}

Binary Resource (e.g. image, PDF)

{
  "contents": [
    {
      "uri": "screen://localhost/display1",
      "mimeType": "image/png",
      "blob": "iVBORw0KGgoAAAANSUhEUgAAA..."
    }
  ]
}

Clients can choose how and when to inject these into the model’s prompt, depending on MIME type and length.

Real-Time Updates

Resources aren’t static—they can change. MCP supports subscriptions to keep context fresh.

List Updates

If the list of resources changes, the server can notify the client with:

notifications/resources/list_changed

Useful when new logs, files, or endpoints become available.

Content Updates

Clients can subscribe to specific resource URIs:

resources/subscribe

When the resource changes, the server sends:

notifications/resources/updated

This is ideal for live logs, dashboards, or real-time documents.

Security Best Practices

Exposing resources to models requires careful control. MCP includes flexible patterns for securing access:

Best Practices for Server Developers

Validate all URIs: No open file reads!
Whitelist paths or endpoints for file access
Use descriptive names and MIME types to help clients filter content
Provide helpful descriptions for the LLM and user
Support URI templates for scalable access
Audit access and subscriptions
Avoid leaking secrets in content or metadata

Example: Safe Log Server

server.setRequestHandler(ListResourcesRequestSchema, async () => {
  return {
    resources: [
      {
        uri: "file:///logs/app.log",
        name: "App Logs",
        mimeType: "text/plain"
      }
    ]
  };
});

server.setRequestHandler(ReadResourceRequestSchema, async (request) => {
  const uri = request.params.uri;

  if (!uri.startsWith("file:///logs/")) {
    throw new Error("Access denied");
  }

  const content = await readFile(uri); // Add sanitization here
  return {
    contents: [{
      uri,
      mimeType: "text/plain",
      text: content
    }]
  };
});

Why Resources Matter for AI Agents

LLMs are context-hungry. They reason better when they have:

Real-time logs
Source code
System metrics
API responses

By serving these as resources, MCP gives agents the data they need—on demand, with full user control, and without bloating prompt templates.

Recap: Resources at a Glance

Feature Description
URI-based identifiers Unique path to each piece of content
Text & binary support Suitable for logs, images, PDFs, etc.
Dynamic templates Construct URIs on the fly
Real-time updates Subscriptions for changing content
Secure access patterns URI validation, MIME filtering, whitelisting

Coming Up Next: Tools in MCP — Giving LLMs the Power to Act

So far, we’ve shown how MCP feeds models with data. But what if we want the model to take action?

In the next post, we’ll explore tools in MCP:

How LLMs call functions safely
Tool schemas and invocation patterns
Real-world examples: shell commands, API calls, and more

A Journey from AI to LLMs and MCP - 7 - Under the Hood — The Architecture of MCP and Its Core Components

Alex Merced — Fri, 11 Apr 2025 09:00:00 GMT

A Journey from AI to LLMs and MCP - 7 - Under the Hood — The Architecture of MCP and Its Core Components

Free Resources

In our last post, we introduced the Model Context Protocol (MCP) as a standard way to connect AI models and agents to tools, data, and workflows—much like how the Apache Iceberg REST protocol brings interoperability to data engines.

Now it’s time to open the black box.

In this post, we’ll break down:

The architecture of MCP
The responsibilities of hosts, clients, and servers
The message lifecycle and transport layers
How tools, resources, and prompts plug into the system

By the end, you’ll understand how MCP enables secure, modular communication between LLMs and the systems they need to work with.

Big Picture: How MCP Fits Together

MCP follows a client-server architecture that enables many-to-many connections between models and systems.

Here’s the high-level setup:

+------------------------+      +--------------------+
|    Claude Desktop      |      |      Web IDE       |
| (Host + MCP Client)    |      | (Host + MCP Client)|
+------------------------+      +--------------------+
             |                         |
             |     MCP Protocol        |
             |                         |
             v                         v
+------------------------+    +---------------------------+
|   Local Tool Server    |    |     Cloud API Server      |
| (Exposes tools/resources)|  | (Exposes prompts/tools)   |
+------------------------+    +---------------------------+

Each host runs one or more clients, which connect to independent MCP servers exposing functionality in a standardized format.

Key Concepts

Let’s look at the core components that make this work.

1. Hosts

Hosts are the applications that run the LLM (e.g. Claude Desktop, VS Code extension, custom browser app). They manage:

The model interaction (LLM prompts and completions)
UI and user input
A registry of connected clients

A host might display tools in a sidebar, allow users to pick files (resources), or visualize prompts in a command palette.

2. Clients

An MCP client lives inside a host and connects to a single MCP server. It handles:

Transport layer setup (e.g. stdio or HTTP/SSE)
Message exchange (requests, notifications, etc.)
Proxying server capabilities to the host/model

Each client maintains a 1:1 connection with one server.

3. Servers

Servers expose real-world capabilities using the MCP spec. They can:

Serve resources (files, logs, database records)
Define and execute tools
Offer reusable prompts
Request sampling (LLM completions)

Servers can run locally (e.g. on your machine) or remotely (e.g. in a cloud API gateway), and can be implemented in any language (Python, TypeScript, C#, etc.).

Message Lifecycle in MCP

MCP uses a JSON-RPC 2.0 message format to communicate between clients and servers. All communication flows through a structured lifecycle:

1. Initialization

Before communication starts:

Client sends an initialize request
Server responds with capabilities
Client sends an initialized notification

This sets up feature negotiation and version compatibility.

2. Message Types

Type	Description
Request	A message expecting a response (e.g. `tools/call`)
Response	Result from a request (e.g. tool output)
Notification	One-way message with no response expected
Error	Sent when a request fails or is invalid

Each message is wrapped in a transport layer (more on that next).

Transport Layer — How Messages Move

MCP supports multiple transport mechanisms:

Stdio Transport

Uses standard input/output
Ideal for local tools and scripts
Simple, reliable, and works well with command-line tools

HTTP + SSE Transport

Uses HTTP POST for client-to-server messages
Uses Server-Sent Events (SSE) for real-time server-to-client updates
Useful for remote or cloud-based servers

All transports carry JSON-RPC messages and follow the same protocol semantics.

MCP Capabilities

MCP defines a small number of core capabilities, each with its own request/response patterns.

Resources

Servers can expose structured data like:

Files
Logs
API responses
Screenshots or binary data

Clients can:

List available resources
Read their contents
Subscribe to updates (e.g. file changes)

Tools

Servers define callable functions that agents can invoke. Each tool has:

A name
Description
JSON schema for inputs
Output format (text or structured)

Tools are model-controlled, meaning the LLM can decide which tool to use based on context.

Prompts

Servers can expose reusable prompt templates with:

Named arguments
Context bindings (e.g. resources)
Multi-step workflows

Prompts are user-controlled, meaning users select when to run them.

Sampling

Servers can ask the host model for completions:

Specify conversation history and preferences
Include system prompt and context
Receive structured completions (text, image, etc.)

This allows server-side workflows to request natural language responses from the model in real time.

Security and Isolation

MCP provides strong boundaries between components:

Hosts control what clients and models can see
Servers expose only the capabilities they choose
Clients can sandbox or restrict tool access
Sampling keeps users in control of what prompts and completions occur

This makes MCP suitable for sensitive environments like IDEs, enterprise apps, and privacy-conscious tools.

Why This Architecture Matters

By standardizing communication between LLMs and tools:

You can plug a new tool into your environment without modifying your agent
You can build servers once and use them across different LLM clients (Claude, custom, etc.)
You get clear separation of concerns: tools, data, and models are independently managed

🔮 Coming Up Next: Resources in MCP — Serving Relevant Data Securely

In the next post, we’ll zoom in on the Resources capability:

How to structure resources
How models use them
Real-world use cases: logs, code, documents, screenshots

Journey from AI to LLMs and MCP - 6 - Enter the Model Context Protocol (MCP) — The Interoperability Layer for AI Agents

Alex Merced — Thu, 10 Apr 2025 09:00:00 GMT

Free Resources

We’ve spent the last few posts exploring the growing power of AI agents—how they can reason, plan, and take actions across complex tasks. And we’ve looked at the frameworks that help us build these agents. But if you’ve worked with them, you’ve likely hit a wall:

Hardcoded toolchains
Limited to a specific LLM provider
No easy way to share tools or data between agents
No consistent interface across clients

What if we had a standard that let any agent talk to any data source or tool, regardless of where it lives or what it’s built with?

That’s exactly what the Model Context Protocol (MCP) brings to the table.

And if you’re from the data engineering world, MCP is to AI agents what the Apache Iceberg REST protocol is to analytics:

A universal, pluggable interface that enables many clients to interact with many servers—without tight coupling.

What Is the Model Context Protocol (MCP)?

MCP is an open protocol that defines how LLM-powered applications (like agents, IDEs, or copilots) can access context, tools, and actions in a standardized way.

Think of it as the "interface layer" between:

Clients: LLMs or AI agents that need context and capabilities
Servers: Local or remote services that expose data, tools, or prompts
Hosts: The environment where the LLM runs (e.g., Claude Desktop, a browser extension, or an IDE plugin)

It defines a common language for exchanging:

Resources (data the model can read)
Tools (functions the model can invoke)
Prompts (templates the user or model can reuse)
Sampling (ways servers can request completions from the model)

This allows you to plug in new capabilities without rearchitecting your agent or retraining your model.

🧱 How MCP Mirrors Apache Iceberg’s REST Protocol

Let’s draw the parallel:

Concept	Apache Iceberg REST	Model Context Protocol (MCP)
Standardized API	REST endpoints for table ops	JSON-RPC messages for context/tools
Decouples client/server	Any engine ↔ any Iceberg catalog	Any LLM/agent ↔ any tool or data backend
Multi-client support	Spark, Trino, Flink, Dremio	Claude, custom agents, IDEs, terminals
Pluggable backends	S3, HDFS, Minio, Pure Storage, GCS	Filesystem, APIs, databases, web services
Interoperable tooling	REST = portable across ecosystems	MCP = portable across LLM environments

Just as Iceberg REST made it possible for Dremio to talk to a table created in Snowflake, MCP allows a tool exposed in Python on your laptop to be used by an LLM in Claude Desktop, a VS Code agent, or even a web-based chatbot.

🔁 MCP in Action — A Real-World Use Case

Imagine this workflow:

You’re coding in an IDE powered by an AI assistant
The model wants to read your logs and run some shell scripts
Your data lives locally, and your tools are custom-built in Python

With MCP:

The IDE (host) runs an MCP client
Your Python tool is exposed via an MCP server
The AI assistant (client) calls your custom “tail logs” tool
The results are streamed back, all through the standardized protocol

And tomorrow, you could replace that assistant with a different model or switch to a browser-based environment—and everything would still work.

The Core Components of MCP

Let’s break down the architecture:

1. Hosts

These are environments where the LLM application lives (e.g., Claude Desktop, your IDE). They manage connections to MCP clients.

2. Clients

Embedded in the host, each client maintains a connection to a specific server. It speaks MCP’s message protocol and exposes capabilities upstream to the model.

3. Servers

Programs that expose capabilities like:

resources/list and resources/read
tools/list and tools/call
prompts/list and prompts/get
sampling/createMessage (to request completions from the model)

Servers can live anywhere: locally on your machine, behind an API, or running in a cloud environment.

What Can MCP Servers Do?

Expose local or remote files (logs, documents, screenshots, live data)
Define tools for executing business logic, running commands, or calling APIs
Provide reusable prompt templates
Request completions from the host model (sampling)

And all of this is done in a protocol-agnostic, secure, pluggable format.

Why This Matters

With MCP, we finally get interoperability in the AI stack—a shared interface layer between:

LLMs and tools
Agents and environments
Models and real-world data

It gives us:

Modularity: Swap out components without breaking workflows
Reusability: Build once, use everywhere
Security: Limit what models can see and do through capabilities
Observability: Track how tools are used and what context is passed
Language-agnostic integration: Servers can be written in Python, JavaScript, C#, and more

In short, MCP helps you go from monolithic, tangled agents to modular, composable AI systems.

What’s Next: Diving Deeper into MCP Internals

In the next few posts, we’ll dig into each part of MCP:

Message formats and lifecycle
How resources and tools are structured
Sampling, prompts, and real-time feedback loops
Best practices for building your own MCP server

A Journey from AI to LLMs and MCP - 5 - AI Agent Frameworks — Benefits and Limitations

Alex Merced — Wed, 09 Apr 2025 09:00:00 GMT

Free Resources

In our last post, we explored what makes an AI agent different from a traditional LLM—memory, tools, reasoning, and autonomy. These agents are the foundation of a new generation of intelligent applications.

But how are these agents built today?

Enter agent frameworks—open-source libraries and developer toolkits that let you create goal-driven AI systems by wiring together models, memory, tools, and logic. These frameworks are enabling some of the most exciting innovations in the AI space... but they also come with trade-offs.

In this post, we’ll dive into:

What AI agent frameworks are
The most popular frameworks available today
The benefits they offer
Where they fall short
Why we need something more modular and flexible (spoiler: MCP)

What Is an AI Agent Framework?

An AI agent framework is a development toolkit that simplifies the process of building LLM-powered systems capable of reasoning, acting, and learning in real time. These frameworks abstract away much of the complexity involved in working with large language models (LLMs) by bundling together key components like memory, tools, task planning, and context management.

Agent frameworks shift the focus from "generating text" to "completing goals." They let developers orchestrate multi-step workflows where an LLM isn't just answering questions but taking action, executing logic, and retrieving relevant data.

Memory

Memory in AI agents refers to how information from past interactions is stored, retrieved, and reused. This can be split into two primary types:

Short-term memory: Keeps track of the current conversation or task state. Usually implemented as a conversation history buffer or rolling context window.
Long-term memory: Stores past interactions, facts, or discoveries for reuse across sessions. Typically backed by:
- A vector database (e.g., Pinecone, FAISS, Weaviate)
- Embedding models that turn text into numerical vectors
- A retrieval layer that finds the most relevant memories using similarity search

Under the hood:

Text is embedded into a vector representation (via models like OpenAI’s text-embedding-ada-002)
These vectors are stored in a database
When new input arrives, it’s embedded and compared to stored vectors
Top matches are fetched and injected into the LLM prompt as background context

Tools

Tools are external functions that the agent can invoke to perform actions or retrieve live information. These can include:

Calling an API (e.g., weather, GitHub, SQL query)
Executing a shell command or script
Reading a file or database
Sending a message or triggering an automation

Frameworks like LangChain, AutoGPT, and Semantic Kernel often use JSON schemas to define tool inputs and outputs. LLMs "see" tool descriptions and decide when and how to invoke them.

Under the hood:

Each tool is registered with a name, description, and parameter schema
The LLM is given a list of available tools and their specs
When the LLM "decides" to use a tool, it returns a structured tool call (e.g., {"name": "search_docs", "args": {"query": "sales trends"}})
The framework intercepts the call, executes the corresponding function, and feeds the result back to the model

This allows the agent to "act" on the world, not just describe it.

🧠 Reasoning and Planning

Reasoning is what enables agents to:

Decompose goals into steps
Decide what tools or memory to use
Track intermediate results
Adjust their strategy based on feedback

Frameworks often support:

React-style loops: Reasoning + action → observation → repeat
Planner-executor separation: One model plans, another carries out steps
Task graphs: Nodes (LLM calls, tools, decisions) arranged in a DAG

Under the hood:

The LLM is prompted to plan tasks using a scratchpad (e.g., "Thought → Action → Observation")
The agent parses the output to decide the next step
Control flow logic (loops, retries, branches) is often implemented in code, not by the model

This turns the agent into a semi-autonomous problem-solver, not just a one-shot prompt engine.

🧾 Context Management

Context management is about deciding what information gets passed into the LLM prompt at any given time. This is critical because:

Token limits constrain how much data can be included
Irrelevant information can degrade model performance
Sensitive data must be filtered for security and compliance

Frameworks handle context by:

Selecting relevant memory or documents via vector search
Condensing history into summaries
Prioritizing inputs (e.g., task instructions, user preferences, retrieved data)
Inserting only high-signal content into the prompt

Under the hood:

Context is assembled as structured messages (usually in OpenAI or Anthropic chat formats)
Some frameworks dynamically prune, summarize, or chunk data to fit within model limits
Smart caching or pagination may be used to maintain continuity across long sessions

Agent frameworks abstract complex functionality into composable components:

Capability	What It Does	How It Works Under the Hood
Memory	Recalls past interactions and facts	Vector embeddings, similarity search, context injection
Tools	Executes real-world actions	Function schemas, LLM tool calls, output feedback loop
Reasoning	Plans steps, decides next action	Thought-action-observation loops, scratchpads
Context Mgmt	Curates what the model sees	Dynamic prompt construction, summarization, filtering

Together, these allow developers to build goal-seeking agents that work across domains—analytics, support, operations, creative work, and more.

Agent frameworks provide the scaffolding. LLMs provide the intelligence.

Popular AI Agent Frameworks

Let’s look at some of the leading options:

LangChain

Language: Python, JavaScript
Strengths:
- Large ecosystem of components
- Support for chains, tools, memory, agents
- Integrates with most major LLMs, vector DBs, and APIs
Limitations:
- Can become overly complex
- Boilerplate-heavy for simple tasks
- Hard to reason about internal agent state

AutoGPT / BabyAGI

Language: Python
Strengths:
- Fully autonomous task execution loops
- Goal-first architecture (recursive reasoning)
Limitations:
- Unpredictable behavior ("runaway agents")
- Tooling and error handling are immature
- Not production-grade (yet)

Semantic Kernel (Microsoft)

Language: C#, Python
Strengths:
- Enterprise-ready tooling
- Strong integration with Microsoft ecosystems
- Planner APIs and plugin system
Limitations:
- Steeper learning curve
- Limited community and examples
- More opinionated structure

CrewAI / MetaGPT

Language: Python
Strengths:
- Multi-agent collaboration
- Role-based task assignment
Limitations:
- Heavy on orchestration
- Still early in maturity
- Debugging agent interactions is hard

Benefits of Using an Agent Framework

These tools have unlocked new possibilities for developers building AI-powered workflows. Let’s summarize the major benefits:

Benefit	Description
Abstractions for Tools	Call APIs or local functions directly from within agent flows
Built-in Memory	Manage short-term context and long-term recall without manual prompt engineering
Modular Design	Compose systems using interchangeable components
Planning + Looping	Support multi-step task execution with feedback loops
Rapid Prototyping	Build functional AI assistants quickly with reusable components

In short: agent frameworks supercharge developer productivity when working with LLMs.

Where Agent Frameworks Fall Short

Despite all their strengths, modern agent frameworks share some core limitations:

1. Tight Coupling to Models and Providers

Most frameworks are tightly bound to OpenAI, Anthropic, or Hugging Face models. Switching providers—or supporting multiple—is complex and risky.

Want to try Claude instead of GPT-4? You might need to refactor your entire chain.

2. Context Management Is Manual and Error-Prone

Choosing what context to pass to the LLM (memory, docs, prior results) is often left to the developer. It’s:

Hard to debug
Easy to overrun token limits
Non-standardized

3. Lack of Interoperability

Most frameworks don’t play well together. Tools, memory stores, and prompt logic often live in their own silos.

You can’t easily plug a LangChain tool into a Semantic Kernel workflow.

4. Hard to Secure and Monitor

Giving agents tool access (e.g., shell commands, APIs) is powerful but risky:

No standard for input validation
No logging/auditing for tool usage
Few controls for human-in-the-loop approvals

5. Opaque Agent Logic

Agents often make decisions that are hard to trace or debug. Why did the agent call that tool? Why did it loop forever?

The Missing Layer: Standardized Context + Tool Protocols

We need a better abstraction layer—something that:

Decouples LLMs from the tools and data they use
Allows agents to access secure, structured resources
Enables modular, composable agents across languages and platforms
Works with any client, model, or provider

That’s where the Model Context Protocol (MCP) comes in.

What’s Next: Introducing the Model Context Protocol (MCP)

In the next post, we’ll explore:

What MCP is
How it enables secure, flexible agent architectures
Why it's the “USB-C port” for LLMs and tools

We’ll walk through the architecture and show how MCP solves many of the problems outlined in this post.

A Journey from AI to LLMs and MCP - 4 - What Are AI Agents — And Why They're the Future of LLM Applications

Alex Merced — Tue, 08 Apr 2025 09:00:00 GMT

Free Resources

We’ve explored how Large Language Models (LLMs) work, and how we can improve their performance with fine-tuning, prompt engineering, and retrieval-augmented generation (RAG). These enhancements are powerful—but they’re still fundamentally stateless and reactive.

To build systems that act with purpose, adapt over time, and accomplish multi-step goals, we need something more.

That “something” is the AI Agent.

In this post, we’ll explore:

What AI agents are
How they differ from LLMs
What components make up an agent
Real-world examples of agent use
Why agents are a crucial next step for AI

What Is an AI Agent?

At a high level, an AI agent is an autonomous or semi-autonomous system built around an LLM, capable of:

Observing its environment (inputs, tools, data)
Reasoning or planning
Taking actions
Learning or adapting over time

LLMs generate responses, but agents make decisions. They don’t just answer; they think, decide, and act.

Think of the difference between a calculator and a virtual assistant. One gives answers. The other gets things done.

The Core Ingredients of an AI Agent

Let’s break down what typically makes up an agentic system:

1. LLM Core

The brain of the operation. Handles natural language understanding and generation.

2. Tools / Actions

Agents can execute external commands, like calling APIs, querying databases, or running code.

3. Memory

Persistent memory lets agents recall previous interactions, facts, or task states.

4. Planner / Executor Logic

This is where agents shine. They can:

Break down complex goals into subtasks
Decide which tools or steps to take
Loop, retry, or adapt based on results

5. Context Manager

Decides what information (memory, documents, tool results) gets included in each LLM prompt.

LLM vs AI Agent — Key Differences

Capability	LLM	AI Agent
Input	Prompt	Prompt + tools + state
Memory	Ephemeral (context)	Persistent (via external memory)
Reasoning	Single-shot	Multi-step planning
Action-taking	No	Yes (tools, APIs, workflows)
Autonomy	None	Optional (user- or goal-directed)
Adaptability	Static behavior	Dynamic, can learn from feedback

LLMs are the engine. Agents are the vehicle.

Examples of AI Agents in the Wild

Let’s explore how AI agents are already showing up in real-world applications:

1. Developer Copilots

Tools like GitHub Copilot or Cursor act as coding assistants, not just autocomplete engines. They:

Read your project files
Ask clarifying questions
Suggest multi-line changes
Run code against test cases

2. Document Q&A Assistants

Instead of just answering questions, agents:

Search relevant documents
Summarize findings
Ask follow-up questions
Offer next actions (e.g., generate reports)

3. Research Agents

Given a broad prompt like “summarize recent news on AI regulation,” agents:

Plan a research strategy
Browse the web or internal data
Synthesize and refine results
Ask for confirmation before continuing

🔄 Agents Enable Autonomy and Feedback Loops

Unlike plain LLMs, agents can:

Use tools to gather more info
Loop on tasks until a condition is met
Store and recall what they’ve seen
Chain multiple steps together

For example:

Task: Schedule a meeting with Alice

Agent:

Search calendar availability
Find Alice’s preferred times
Draft an email proposal
Wait for response
Reschedule if needed

That’s not a single LLM prompt—that’s an intelligent system managing an evolving task.

How Are Agents Built Today?

A number of popular AI agent frameworks have emerged:

LangChain: Modular orchestration of LLMs, tools, and memory
AutoGPT: Autonomous task completion with iterative planning
Semantic Kernel: Microsoft’s framework for embedding LLMs into software
CrewAI / MetaGPT: Multi-agent systems with defined roles

These frameworks let developers prototype powerful workflows, but they come with challenges—especially around complexity, tool integration, and portability.

We’ll explore those challenges in the next post.

Limitations of Today’s Agent Implementations

While agents are promising, current frameworks have some limitations:

Tight coupling to specific models or tools
Difficult interoperability between agent components
Context juggling: hard to manage what the model sees
Security and control: risk of unsafe tool access
Hard to debug: agents can go rogue or get stuck in loops

To address these, we need standardization—a modular way to plug in data, tools, and models securely and flexibly.

That’s where the Model Context Protocol (MCP) enters the picture.

Coming Up Next: AI Agent Frameworks — Benefits and Limitations

In our next post, we’ll explore:

How modern agent frameworks work
What they enable (and where they fall short)
The missing layer that MCP provides

A Journey from AI to LLMs and MCP - 3 - Boosting LLM Performance — Fine-Tuning, Prompt Engineering, and RAG

Alex Merced — Mon, 07 Apr 2025 09:00:00 GMT

Free Resources

In our last post, we explored how LLMs process text using embeddings and vector spaces within limited context windows. While LLMs are powerful out-of-the-box, they aren’t perfect—and in many real-world scenarios, we need to push them further.

That’s where enhancement techniques come in.

In this post, we’ll walk through the three most popular and practical ways to boost the performance of Large Language Models (LLMs):

Fine-tuning
Prompt engineering
Retrieval-Augmented Generation (RAG)

Each approach has its strengths, trade-offs, and ideal use cases. By the end, you’ll know when to use each—and how they work under the hood.

1. Fine-Tuning — Teaching the Model New Tricks

Fine-tuning is the process of training an existing LLM on custom datasets to improve its behavior on specific tasks.

How it works:

You take a pre-trained model (like GPT or LLaMA).
You feed it new examples in a structured format (instructions + completions).
The model updates its internal weights based on this new data.

Think of it like giving the model a focused education after it’s graduated from a general AI university.

When to use it:

You want a custom assistant that uses your company’s voice
You need the model to perform a specialized task (e.g., legal analysis, medical diagnostics)
You have recurring, structured inputs that aren’t handled well with prompting alone

Trade-offs:

Pros	Cons
Highly accurate for specific tasks	Expensive (compute + time)
Reduces prompt complexity	Risk of overfitting or forgetting
Works well offline or locally	Not ideal for frequently changing data

Fine-tuning is powerful, but it’s not always the first choice—especially when you need flexibility or real-time knowledge.

2. Prompt Engineering — Speaking the Model’s Language

Sometimes, you don’t need to retrain the model—you just need to talk to it better.

Prompt engineering is the art of crafting inputs that guide the model to behave the way you want. It’s fast, flexible, and doesn’t require model access.

Prompting patterns:

Zero-shot prompting: Just ask a question

“Summarize this article.”
Few-shot prompting: Show examples

“Here’s how I want you to respond…”
Chain-of-Thought (CoT): Encourage reasoning

“Let’s think step by step…”

Tools and techniques:

Templates: Reusable format strings with variables
Constraints: “Answer in JSON” or “Limit to 100 words”
Personas: “You are a helpful legal assistant...”
System prompts (where supported): Define role and tone

When to use it:

You’re working with a hosted LLM (OpenAI, Anthropic, etc.)
You want to avoid infrastructure and cost overhead
You need to quickly iterate and improve outcomes

Trade-offs:

Pros	Cons
Fast to test and implement	Sensitive to wording
Doesn’t require model access	Can be brittle or unpredictable
Great for prototyping	Doesn’t scale well for complex logic

Prompt engineering is like UX for AI—small changes in input can completely change the output.

3. Retrieval-Augmented Generation (RAG) — Give the Model Real-Time Knowledge

RAG is a game-changer for context-aware applications.

Instead of cramming all your knowledge into a model, RAG retrieves relevant information at runtime and includes it in the prompt.

How it works:

User sends a query
System runs a semantic search over a vector database
Top-matching documents are inserted into the prompt
The LLM generates a response using both query + retrieved context

This gives you dynamic, real-time access to external knowledge—without retraining.

Typical RAG architecture:

User → Query → Vector Search (Embeddings) → Top K Documents → LLM Prompt → Response

Use case examples:

Chatbots that answer questions from company docs
Developer copilots that can search codebases
LLMs that read log files, support tickets, or PDFs

Trade-offs:

Pros	Cons
Real-time access to changing data	Adds latency due to search layer
No need to retrain the model	Requires infrastructure (DB + search)
Keeps context windows lean	Needs good chunking & ranking logic

With RAG, your LLM becomes a smart interface to your data—not just the internet.

Choosing the Right Enhancement Technique

Here’s a quick cheat sheet to help you choose:

Goal	Best Technique
Specialize a model on internal tasks	Fine-tuning
Guide output or behavior flexibly	Prompt engineering
Inject dynamic, real-time knowledge	Retrieval-Augmented Gen

Often, the best systems combine these techniques:

Fine-tuned base model
With prompt templates
And external knowledge via RAG

This is exactly what advanced AI agent systems are starting to do—and it’s where we’re heading next.

Recap: Boosting LLMs Is All About Context and Control

Technique	What It Does	Ideal For
Fine-Tuning	Teaches model new behavior	Repetitive, specialized tasks
Prompt Engineering	Crafts effective inputs	Fast prototyping, hosted models
RAG	Adds knowledge dynamically at runtime	Large, evolving, external datasets

Up Next: What Are AI Agents — And Why They’re the Future

Now that we’ve learned how to enhance individual LLMs, the next evolution is combining them with tools, memory, and logic to create AI Agents.

In the next post, we’ll explore:

What makes something an AI agent
How agents orchestrate LLMs + tools
Why they’re essential for real-world use

A Journey from AI to LLMs and MCP - 2 - How LLMs Work — Embeddings, Vectors, and Context Windows

Alex Merced — Sun, 06 Apr 2025 09:00:00 GMT

Free Resources

In our last post, we explored the evolution of AI—from rule-based systems to deep learning—and how Large Language Models (LLMs) like GPT-4 and Claude represent a transformative leap in capability.

But how do these models actually work?

In this post, we’ll peel back the curtain on the inner workings of LLMs. We’ll explore the fundamental concepts that make these models tick: embeddings, vector spaces, and context windows. You’ll walk away with a clearer understanding of how LLMs “understand” language—and what their limits are.

How LLMs Think: It’s All Math Underneath

Despite their fluent text output, LLMs don’t truly "understand" language in the human sense. Instead, they operate on numerical representations of text, using vast networks of mathematical weights to predict the next word in a sequence.

The key mechanism behind this: transformers.

Transformers revolutionized NLP by allowing models to weigh the relevance of each word in a sentence—attention mechanisms—instead of processing words one-by-one like RNNs.

Here’s the simplified flow:

Text is tokenized (split into chunks)
Tokens are converted into embeddings (vectors)
Those vectors pass through layers of attention to capture meaning
The model generates the next token based on probability

But what are these embeddings and why do they matter?

Embeddings: From Words to Numbers

Before an LLM can do anything with language, it must convert words into numbers it can operate on.

That’s where embeddings come in.

What is an embedding?

An embedding is a high-dimensional vector (think: a long list of numbers) that represents the meaning of a word or phrase.

Words with similar meanings have similar embeddings.

For example:

Embedding("dog") ≈ Embedding("puppy") Embedding("Paris") ≈ Embedding("London")

These vectors live in an abstract vector space, where distance encodes similarity.

LLMs use embeddings not just for input, but throughout every layer of their neural network to understand relationships, context, and meaning.

Vector Search and Semantic Understanding

Because embeddings encode meaning, they’re also incredibly useful for semantic search.

Instead of matching exact words (like keyword search), vector search compares embeddings to find text that’s conceptually similar.

For example:

Query: "How do I fix a leaking pipe?"
Match: "Plumbing repair for minor water leaks"

Even though the words don’t overlap, the meaning does—and that’s what embeddings capture.

This is the foundation for many powerful AI techniques like:

Document similarity
Retrieval-Augmented Generation (RAG) (more on this in Blog 3)
Context injection from external data sources

Context Windows: The Model’s Working Memory

Another crucial concept in LLMs is the context window—the maximum number of tokens the model can “see” at once.

Every input to an LLM gets broken into tokens, and the model has a limited capacity for how many tokens it can process per request.

Model	Max Context Window
GPT-3.5	4,096 tokens (~3,000 words)
GPT-4 Turbo	Up to 128,000 tokens
Claude 3 Opus	Up to 200,000 tokens

If you go over the limit, you’ll need to:

Truncate input (losing information)
Summarize
Use techniques like RAG or memory management

TL;DR: The larger the context window, the more the model can “remember” during a conversation or task.

Limitations of Embeddings and Context Windows

Even though LLMs are powerful, they come with trade-offs:

Embedding limitations:

Don’t always reflect nuanced context (e.g., sarcasm, tone)
Fixed dimensionality: can’t represent everything
Require separate handling for different modalities (text vs images)

Context window limitations:

Long documents may get truncated or ignored
Memory is not persistent—everything resets after a session unless you manually re-include previous context
More tokens = higher latency and cost

These limits are precisely why so much effort goes into enhancing LLMs through fine-tuning, retrieval systems, and smarter prompt engineering.

We’ll dive into that next.

Recap: Key Concepts from This Post

Concept	What It Is	Why It Matters
Embeddings	Vector representations of tokens/text	Enable semantic understanding & search
Vector Space	Mathematical space where embeddings live	Allows similarity comparison & clustering
Context Window	Max token size per LLM input	Defines how much the model can “see”
Attention	Weighs token relationships dynamically	Enables context awareness in LLMs

🔮 Up Next: Making LLMs Smarter with Fine-Tuning, Prompt Engineering, and RAG

In our next post, we’ll show how to enhance LLM performance using proven techniques:

Fine-tuning
Prompt engineering
Retrieval-Augmented Generation (RAG)

These strategies help you move beyond limitations—and get the most out of your models.

A Journey from AI to LLMs and MCP - 1 - What Is AI and How It Evolved Into LLMs

Alex Merced — Sat, 05 Apr 2025 09:00:00 GMT

Free Resources

Artificial Intelligence (AI) has become the defining technology of the decade. From chatbots to code generators, from self-driving cars to predictive text—AI systems are everywhere. But before we dive into the cutting-edge world of large language models (LLMs), let’s rewind and understand where this all began.

This post kicks off our 10-part series exploring how AI evolved into LLMs, how to enhance their capabilities, and how the Model Context Protocol (MCP) is shaping the future of intelligent, modular agents.

🧠 A Brief History of AI

The term "Artificial Intelligence" was coined in 1956, but the idea has been around even longer—think mechanical automatons and Alan Turing’s famous question: "Can machines think?"

AI development has gone through several distinct waves:

1. Symbolic AI (1950s–1980s)

Also known as "Good Old-Fashioned AI," symbolic systems were rule-based. Think expert systems, logic programming, and hand-coded decision trees. These systems could play chess or diagnose medical conditions—if you wrote enough rules.

Limitations: Rigid, brittle, and poor at handling ambiguity.

2. Machine Learning (1990s–2010s)

Instead of coding rules manually, we trained models to recognize patterns from data. Algorithms like decision trees, support vector machines, and early neural networks emerged.

This era gave us:

Spam filters
Fraud detection
Recommendation engines

But while powerful, these models still had a hard time with natural language and context.

3. Deep Learning (2010s–Now)

With more data, better algorithms, and stronger GPUs, neural networks started outperforming traditional methods. Deep learning led to breakthroughs in:

Image recognition (CNNs)
Speech recognition (RNNs, LSTMs)
Language understanding (Transformers)

And that brings us to the latest evolution...

🧬 Enter LLMs: The Rise of Language-First AI

Large Language Models (LLMs) like GPT-4, Claude, and Gemini aren’t just another step in AI—they represent a leap. Trained on massive text corpora using transformer architectures, these models can:

Write essays and poems
Generate and debug code
Translate between languages
Answer complex questions

All by predicting the next word in a sentence.

But what makes LLMs so powerful?

🏗️ LLMs Are More Than Just Big Neural Nets

At their core, LLMs are massive deep learning models that turn tokens (words/pieces of words) into vectors (mathematical representations). Through billions of parameters, they learn the structure of language and the latent meaning within it.

Key components:

Tokenization: Breaking input into chunks the model can process
Embeddings: Mapping tokens to vector space
Attention Mechanisms: Letting the model focus on relevant parts of the input
Context Window: A memory buffer for how much input the model can “see”

Popular LLMs:

Model	Provider	Context Window	Notable Feature
GPT-4	OpenAI	Up to 128k	Code + natural language synergy
Claude 3	Anthropic	Up to 200k	Strong at instruction following
Gemini	Google DeepMind	~32k+	Multimodal capabilities

🧩 What LLMs Can (and Can’t) Do

LLMs are versatile and impressive—but they're not magic. Their strengths come with real limitations:

✅ What they’re great at:

Text generation and summarization
Conversational interfaces
Programming assistance
Knowledge retrieval from training data

❌ What they struggle with:

Memory: No persistent memory across sessions
Context limits: Can only “see” a fixed number of tokens
Reasoning: Struggles with complex multi-step logic
Real-time data: Can’t access up-to-date or private information
Action-taking: Can't interact with tools or APIs by default

This is where the next evolution comes in: augmenting LLMs with context, tools, and workflows.

🔮 The Road Ahead: From Models to Modular AI Agents

We’ve gone from rules to learning, from deep learning to LLMs—but we’re not done yet. The future of AI lies in making LLMs do more than just talk. We need to:

Give them memory
Let them interact with data
Enable them to call tools, services, and APIs
Help them make decisions and reason through complex tasks

This brings us to the idea of AI Agents—autonomous systems built on LLMs that can perceive, decide, and act.

🧭 Coming Up Next

In our next post, we’ll explore how LLMs actually work under the hood—digging into embeddings, vector spaces, and how models “understand” language.

Stay tuned.

Building a Basic MCP Server with Python

Alex Merced — Fri, 04 Apr 2025 09:00:00 GMT

Free Resources

If you’ve ever wished you could ask an AI model like Claude to interact with your local files or run custom code—good news: you can. That’s exactly what the Model Context Protocol (MCP) makes possible.

In this tutorial, we’ll walk you through building a beginner-friendly MCP server that acts as a simple template for future projects. You don’t need to be an expert in AI or server development—we’ll explain each part as we go.

Here’s what we’ll build:

A small server using Python and the MCP SDK
Two useful tools that read data from:
- A CSV file (great for spreadsheets and tabular data)
- A Parquet file (a format often used in data engineering and analytics)
A clean folder structure that makes it easy to add new tools or features later
A working connection to Claude for Desktop, so you can ask things like:

“Summarize the contents of my data file”
“How many rows and columns are in this CSV?”

Why Start Here?

This blog is perfect for you if:

You’ve heard about Claude and want to connect it to your own tools or data
You’re curious about MCP and want to see how it works in practice
You’d like a solid starting point for building more advanced tool servers later

We’ll use plain Python and some common libraries like pandas, with no web frameworks or deployment complexity. Everything will run locally on your machine.

By the end, you’ll have a fully working local MCP server and a better understanding of how to make AI tools that go beyond text prediction—and actually do useful work.

Let’s get started!

What Is MCP (and Why Should You Care)?

Let’s break this down before we start writing code.

MCP stands for Model Context Protocol. It’s a way to let apps like Claude for Desktop securely interact with external data and custom tools that you define.

Think of it like building your own mini API—but instead of exposing it to the whole internet, you’re exposing it to an AI assistant on your machine.

With MCP, you can:

Let Claude read a file or query a database
Create tools that do useful things (like summarize a dataset or fetch an API)
Add reusable prompts to guide how Claude behaves in certain tasks

For this project, we’re focusing on tools—the part of MCP that lets you write small Python functions the AI can call.

What We’re Building

Here’s a quick preview of what you’ll end up with:

A local MCP server called mix_server
Two tools: one that reads a CSV file, and one that reads a Parquet file
A clean, modular folder layout so you can keep adding more tools later
A working connection to Claude for Desktop so you can talk to your tools through natural language

Let’s start by setting up your project.

Project Setup (Step-by-Step)

We’ll use uv—a fast, modern Python project manager—to create and manage our environment. It handles dependencies, virtual environments, and script execution, all in one place.

If you’ve used pip or virtualenv before, uv is like both of those combined—but much faster and more ergonomic.

Step 1: Install `uv`

To install uv, run this in your terminal:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then restart your terminal so the uv command is available.

You can check that it's working with:

uv --version

Step 2: Create the Project

Let’s make a new folder for our MCP server:

uv init mix_server
cd mix_server

This creates a basic Python project with a pyproject.toml file to manage dependencies.

Step 3: Set Up a Virtual Environment

We’ll now create a virtual environment for our project and activate it:

uv venv
source .venv/bin/activate

This keeps your dependencies isolated from the rest of your system.

Step 4: Add Required Dependencies

We’re going to install three key packages:

mcp[cli]: The official MCP SDK and command-line tools
pandas: For reading CSV and Parquet files
pyarrow: Adds support for reading Parquet files via Pandas

Install them using:

uv add "mcp[cli]" pandas pyarrow

This updates your pyproject.toml and installs the packages into your environment.

Step 5: Create a Clean Folder Structure

We’ll use the following layout to stay organized:

mix_server/
│
├── data/                 # Sample CSV and Parquet files
│
├── tools/                # MCP tool definitions
│
├── utils/                # Reusable file reading logic
│
├── server.py             # Creates the Server
├── main.py             # Entry point for the MCP server
└── README.md             # Optional documentation

Create the folders:

mkdir data tools utils
touch server.py

Your environment is now ready. In the next section, we’ll create a couple of small data files to work with—a CSV and a Parquet file—and use them to power our tools.

Creating Sample Data Files

To build our first tools, we need something for them to work with. In this section, we’ll create two simple files:

A CSV file (great for spreadsheets and tabular data)
A Parquet file (a more efficient format used in data engineering)

Both files will contain the same mock dataset—a short list of users. You’ll use these files later when building tools that summarize their contents.

Step 1: Create the `data/` Folder

If you haven’t already created the folder for our data, do it now from your project root:

mkdir data

Step 2: Create a Sample CSV File

Now let’s add a sample CSV file with some fake user data.

Create a new file called sample.csv inside the data/ folder:

data/sample.csv

And paste the following into it:

id,name,email,signup_date
1,Alice Johnson,alice@example.com,2023-01-15
2,Bob Smith,bob@example.com,2023-02-22
3,Carol Lee,carol@example.com,2023-03-10
4,David Wu,david@example.com,2023-04-18
5,Eva Brown,eva@example.com,2023-05-30

This file gives us structured, readable data—perfect for a tool to analyze.

Step 3: Convert the CSV to Parquet

We’ll now create a Parquet version of the same data using Python. This shows how easily you can support both file types in your tools.

Create a short script in the root of your project called generate_parquet.py:

# generate_parquet.py

import pandas as pd

# Read the CSV
df = pd.read_csv("data/sample.csv")

# Save as Parquet
df.to_parquet("data/sample.parquet", index=False)

Run the script:

uv run generate_parquet.py

After this, your data/ folder should look like:

data/
├── sample.csv
└── sample.parquet

What’s the Difference Between CSV and Parquet?

CSV: Simple, human-readable text file. Great for small datasets and quick inspection.
Parquet: A binary, column-based format. Much faster for large datasets and common in analytics pipelines (e.g. with Apache Spark or Dremio).

Supporting both formats makes your tools more flexible, and this example shows how little extra effort it takes.

Next, we’ll write some reusable utility functions that can read these files and return a quick summary of their contents—ready to be wrapped as MCP tools.

Writing Utility Functions to Read CSV and Parquet Files

Now that we have some data to work with, let’s write the core logic to read those files and return a basic summary.

We’re going to put this logic in a separate Python file under a folder called utils/. This makes it easy to reuse across different tools without duplicating code.

Step 1: Create the Utility Module

If you haven’t already created the utils/ folder, do it now:

mkdir utils

Now create a new Python file inside it:

touch utils/file_reader.py

Step 2: Add File Reading Functions

Open utils/file_reader.py and paste in the following code:

# utils/file_reader.py

import pandas as pd
from pathlib import Path

# Base directory where our data lives
DATA_DIR = Path(__file__).resolve().parent.parent / "data"

def read_csv_summary(filename: str) -> str:
    """
    Read a CSV file and return a simple summary.

    Args:
        filename: Name of the CSV file (e.g. 'sample.csv')

    Returns:
        A string describing the file's contents.
    """
    file_path = DATA_DIR / filename
    df = pd.read_csv(file_path)
    return f"CSV file '{filename}' has {len(df)} rows and {len(df.columns)} columns."

def read_parquet_summary(filename: str) -> str:
    """
    Read a Parquet file and return a simple summary.

    Args:
        filename: Name of the Parquet file (e.g. 'sample.parquet')

    Returns:
        A string describing the file's contents.
    """
    file_path = DATA_DIR / filename
    df = pd.read_parquet(file_path)
    return f"Parquet file '{filename}' has {len(df)} rows and {len(df.columns)} columns."

How This Works

We’re using pandas to read both CSV and Parquet files. It’s a well-known data analysis library in Python.
pathlib.Path helps us safely construct file paths across operating systems.

Both functions return a simple string like:

CSV file 'sample.csv' has 5 rows and 4 columns.

This is all the logic our tools will need to start with. Later, if you want to add more advanced summaries—like listing column names or detecting null values—you can expand these functions.

With our utilities ready, we can now expose them as MCP tools—so Claude can actually use them!

Wrapping File Readers as MCP Tools

Now that we’ve written the logic to read and summarize our data files, it’s time to make those functions available to Claude through MCP tools.

What’s an MCP Tool?

An MCP tool is a Python function you register with your MCP server that the AI can call when it needs to take action—like reading a file, querying an API, or performing a calculation.

To register a tool, you decorate the function with @mcp.tool(). Behind the scenes, MCP generates a definition that the AI can see and interact with.

But before we do that, let’s follow a best practice: we’ll define our MCP server instance in one central place, then import it into each file that defines tools. This ensures everything stays clean and consistent.

Step 1: Define the MCP Server Instance

Open your server.py and main.py files (or create it if you haven’t already), and add the following:

# server.py

from mcp.server.fastmcp import FastMCP

# This is the shared MCP server instance
mcp = FastMCP("mix_server")

from server import mcp

# Entry point to run the server
if __name__ == "__main__":
    mcp.run()

This creates a named server called "mix_server" and exposes a simple run command.

Step 2: Create the CSV Tool

Let’s now define our first tool: one that summarizes a CSV file.

Create a new file called csv_tools.py inside the tools/ folder:

touch tools/csv_tools.py

Then add the following:

# tools/csv_tools.py

from server import mcp
from utils.file_reader import read_csv_summary

@mcp.tool()
def summarize_csv_file(filename: str) -> str:
    """
    Summarize a CSV file by reporting its number of rows and columns.

    Args:
        filename: Name of the CSV file in the /data directory (e.g., 'sample.csv')

    Returns:
        A string describing the file's dimensions.
    """
    return read_csv_summary(filename)

Step 3: Create the Parquet Tool

Now let’s do the same for a Parquet file.

Create a file called parquet_tools.py inside the tools/ folder:

touch tools/parquet_tools.py

And add:

# tools/parquet_tools.py

from server import mcp
from utils.file_reader import read_parquet_summary

@mcp.tool()
def summarize_parquet_file(filename: str) -> str:
    """
    Summarize a Parquet file by reporting its number of rows and columns.

    Args:
        filename: Name of the Parquet file in the /data directory (e.g., 'sample.parquet')

    Returns:
        A string describing the file's dimensions.
    """
    return read_parquet_summary(filename)

Step 4: Register the Tools

Since the tools are registered via decorators at import time, we just need to make sure the server.py file imports the tool modules. Add these lines at the top of server.py:

# main.py

from server import mcp

# Import tools so they get registered via decorators
import tools.csv_tools
import tools.parquet_tools

# Entry point to run the server
if __name__ == "__main__":
    mcp.run()

Now, whenever the server runs, it automatically registers all tools via the @mcp.tool() decorators.

Your tools are now live! In the next section, we’ll walk through how to run the server and connect it to Claude for Desktop so you can test them out in natural language.

Running and Testing Your MCP Server with Claude for Desktop

At this point, you’ve built a functional MCP server with two tools: one for reading CSV files and another for Parquet. Now it’s time to bring it to life and connect it to Claude for Desktop, so you can start running your tools using plain English.

Step 1: Run the Server

Let’s start your server locally.

In your project root (where server.py lives), run:

uv run main.py

This starts your MCP server using the tools you defined. You won’t see much output in the terminal just yet—that’s normal. Your server is now waiting for a connection from a client like Claude.

Step 2: Install Claude for Desktop (If You Haven’t Already)

You’ll need Claude for Desktop installed to connect to your server.

Download it here: https://www.anthropic.com/claude

Follow the installation instructions for your operating system

Note: As of now, Claude for Desktop is not available on Linux. If you’re on Linux, skip ahead to the section on building your own MCP client.

Step 3: Configure Claude to Use Your Server

Claude needs to know where to find your MCP server. You’ll do this by editing a small config file on your system.

MacOS / Linux:

Open this file in your code editor (create it if it doesn’t exist):

code ~/Library/Application\ Support/Claude/claude_desktop_config.json

Windows:

The config file is located here:

%APPDATA%\Claude\claude_desktop_config.json

Step 4: Add Your Server to the Config

Paste the following JSON into the file, replacing the "/ABSOLUTE/PATH/..." with the actual full path to your mix_server project folder:

{
  "mcpServers": {
    "mix_server": {
      "command": "uv",
      "args": [
        "--directory",
        "/ABSOLUTE/PATH/TO/mix_server",
        "run",
        "main.py"
      ]
    }
  }

Tip: To find the absolute path:

On Mac/Linux: Run pwd in your terminal

On Windows: Use cd and copy the full path from File Explorer

Make sure uv is in your system PATH, or replace "command": "uv" with the full path to the uv executable.

Step 5: Restart Claude for Desktop

Restart the app, and you should see a new tool icon (hammer) appear in the interface. Click it, and you’ll see your registered tools:

summarize_csv_file
summarize_parquet_file

These can now be called directly by the AI!

Step 6: Try It Out

Now try asking Claude something like:

"Summarize the CSV file named sample.csv."
"How many rows are in sample.parquet?"

Claude will detect the appropriate tool, call your server, and respond with the results—powered by the very Python code you wrote.

Troubleshooting Tips

If things don’t work right away, here are a few things to check:

Make sure your uv run main.py process is running and hasn't crashed
Ensure the file paths in your config JSON are correct
Confirm that your data files (sample.csv, sample.parquet) exist in the /data directory
Check the Claude UI for error messages or tool-loading indicators

You now have a working local AI toolchain powered by MCP! In the final section, we’ll do a quick recap and show how you can build on this template for more powerful tools.

Recap and Next Steps

Congratulations—you just built your first MCP server!

Let’s take a moment to review what you’ve accomplished.

What You Built

By following this guide, you now have a fully working MCP server that:

Uses Python and the official mcp SDK
Reads real data from both CSV and Parquet files
Exposes two custom MCP tools that Claude for Desktop can call:
- summarize_csv_file
- summarize_parquet_file
Follows a clean, modular folder structure
Runs locally using uv and connects seamlessly to Claude for natural language interaction

You also learned how to:

Set up your Python project with uv
Manage dependencies cleanly
Register and expose tools using the @mcp.tool() decorator
Wire everything together with Claude through a simple config file

Where to Go From Here

This project was intentionally simple so you could focus on learning the structure and flow of an MCP server. But this is just the beginning.

Here are a few ideas for extending this template:

1. Add More Advanced Tools

Try building tools that:

Filter rows based on a column value
Return column names or data types
Calculate statistics (mean, median, etc.)

2. Use Resources

Use @mcp.resource() to expose static or dynamic data that Claude can pull into its context before making a decision.

3. Explore Prompts

Create reusable interaction templates with @mcp.prompt() to guide how Claude asks or responds.

4. Add Async Logic

If you’re pulling data from APIs or databases, consider making your tools async using async def—fully supported by FastMCP.

5. Build Your Own Client

Not using Claude? You can write your own MCP-compatible client using the SDK’s ClientSession interface.

Share and Reuse

You now have a template you can reuse for future projects. If you publish it on GitHub, others can fork it, extend it, and learn from it too.

This isn’t just a demo—it’s the foundation of a toolchain where you can define your own AI-powered workflows and expose them to LLMs in a controlled, modular way.

Using Helm with Kubernetes - A Guide to Helm Charts and Their Implementation

Alex Merced — Wed, 19 Feb 2025 09:00:00 GMT

Free Resources

Managing applications in Kubernetes can be complex, requiring multiple YAML files to define resources such as Deployments, Services, ConfigMaps, and Secrets. As applications scale, maintaining and updating these configurations manually becomes cumbersome and error-prone. This is where Helm comes in.

Helm is a package manager for Kubernetes that simplifies deployment by bundling application configurations into reusable, version-controlled Helm charts. With Helm, you can deploy applications with a single command, manage updates seamlessly, and roll back to previous versions if needed.

Why Use Helm?

Simplifies Deployments – Deploy complex applications with a single command instead of managing multiple YAML files.
Parameterization & Reusability – Configure deployments dynamically using values.yaml, making it easy to manage multiple environments (dev, staging, prod).
Version Control & Rollbacks – Helm tracks deployments, allowing you to roll back to previous versions in case of failures.
Dependency Management – Install and manage application dependencies effortlessly.
Integration with CI/CD & GitOps – Automate deployments with tools like ArgoCD, FluxCD, and GitHub Actions.

What You'll Learn in This Guide

In this blog, we’ll cover:

What Helm is and how it works – Understanding its architecture and components.
Installing and configuring Helm – Setting up Helm for your Kubernetes cluster.
Understanding Helm charts – Exploring chart structure, templates, and values.
Writing your own Helm chart – Step-by-step guide to creating a custom chart.
Deploying applications with Helm – Installing, upgrading, and rolling back releases.
Best practices for Helm in production – Security, GitOps integration, and monitoring.

By the end of this guide, you'll have a strong foundation in Helm and be able to deploy, manage, and scale Kubernetes applications efficiently.

Understanding Helm: The Package Manager for Kubernetes

What is Helm?

Helm is a package manager for Kubernetes that helps deploy, configure, and manage applications in a Kubernetes cluster. Instead of manually writing and applying multiple Kubernetes YAML manifests, Helm allows you to package them into reusable Helm Charts, simplifying deployment and maintenance.

Why Use Helm?

Managing Kubernetes resources can become complex, especially when deploying applications with multiple components (Deployments, Services, ConfigMaps, Secrets, etc.). Helm provides several advantages:

Simplifies Deployments – Automates the process of applying multiple YAML files.
Versioning & Rollbacks – Tracks different versions of deployments and allows rollback if necessary.
Parameterization & Reusability – Uses a templating system (values.yaml) to customize deployments.
Dependency Management – Simplifies installing and upgrading application dependencies.
Consistent Configuration Across Environments – Makes it easy to manage different configurations for dev, staging, and production.

How Does Helm Compare to Traditional Kubernetes Manifests?

Feature	Kubernetes YAML Manifests	Helm Charts
Management	Requires manually applying multiple YAML files	Uses a single Helm command
Configuration	Static YAML definitions	Dynamic templating via `values.yaml`
Version Control	Difficult to track changes manually	Built-in versioning & rollback
Reusability	Limited; each deployment needs its own YAML	Reusable and configurable charts
Dependencies	Managed manually	Handled via `requirements.yaml` (deprecated) or `Chart.yaml`

How Helm Works

Helm Components and Architecture

Helm follows a client-only architecture in Helm v3, where it directly interacts with the Kubernetes API server without requiring a backend component like Tiller (which was used in Helm v2). Below are the core components of Helm:

Helm CLI – The command-line interface used to manage Helm charts, releases, and repositories.
Helm Charts – Packaged Kubernetes applications that define resources like Deployments, Services, ConfigMaps, and Secrets.
Helm Repository – A collection of Helm charts stored in a remote or local location (e.g., Artifact Hub).
Helm Release – A deployed instance of a Helm chart, stored as metadata inside the Kubernetes cluster.
Kubernetes API Server – Helm interacts with the Kubernetes API to apply resources as defined in the chart.

Helm Workflow: How Helm Manages Deployments

Fetching Charts – Helm can pull pre-built charts from repositories using helm repo add and helm search repo.
Templating and Rendering – Helm dynamically replaces values in the YAML templates using the values.yaml file before applying them.
Creating a Release – When a Helm chart is installed, Helm assigns it a unique release name and applies the rendered templates to the Kubernetes cluster.
Versioning and Rollbacks – Helm maintains a history of releases, allowing easy upgrades (helm upgrade) and rollbacks (helm rollback).
Uninstalling Releases – Helm can remove all associated Kubernetes resources using helm uninstall.

Helm Command Lifecycle

Command	Purpose
`helm repo add <repo-name> <repo-url>`	Adds a Helm chart repository
`helm search repo <keyword>`	Searches for a chart in repositories
`helm install <release-name> <chart-name>`	Installs a Helm chart and creates a release
`helm list`	Lists all active Helm releases
`helm status <release-name>`	Shows details of a deployed release
`helm upgrade <release-name> <chart-name>`	Upgrades an existing release to a new chart version
`helm rollback <release-name> <revision>`	Rolls back a release to a previous version
`helm uninstall <release-name>`	Deletes a release and removes associated resources

Helm in Action: A Simple Example

Let's say you want to deploy NGINX using Helm. You can do this with a single command:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-nginx bitnami/nginx

This command:

Adds the Bitnami Helm repository.
Installs the NGINX Helm chart from the Bitnami repository.
Creates a Helm release named my-nginx in the cluster.

To check the status of the deployment:

helm list
helm status my-nginx

To uninstall the release:

helm uninstall my-nginx

Installing and Configuring Helm

Before using Helm, you need to install it on your local machine and configure it to work with your Kubernetes cluster. This section will walk through the installation process and initial setup.

Prerequisites

A Kubernetes cluster running locally (e.g., Minikube, Kind) or in the cloud (e.g., AKS, GKE, EKS).
kubectl installed and configured to communicate with your cluster.

Installing Helm

Helm can be installed on macOS, Linux, and Windows using various package managers.

macOS (Using Homebrew)

brew install helm

Linux (Using Script)

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Windows (Using Chocolatey)

choco install kubernetes-helm

Verifying the Installation

After installation, verify that Helm is installed correctly by running:

helm version

You should see output similar to:

version.BuildInfo{Version:"v3.x.x", GitCommit:"...", GitTreeState:"clean", GoVersion:"..."}

Configuring Helm

Adding a Helm Repository

Helm uses repositories to store charts. You can add a popular repository, such as the Bitnami Helm charts, using:

helm repo add bitnami https://charts.bitnami.com/bitnami

To confirm the repository has been added:

helm repo list

Updating Helm Repositories

To fetch the latest charts from all added repositories, run:

helm repo update

Searching for Helm Charts

To search for a specific application within your configured repositories:

helm search repo nginx

Installing a Helm Chart

Once Helm is set up, you can deploy an application. For example, to deploy NGINX using the Bitnami Helm chart:

helm install my-nginx bitnami/nginx

This will:

Download the NGINX chart.
Deploy the necessary Kubernetes resources.
Assign the release name my-nginx.

Checking the Installation

List all active Helm releases:

helm list

Check the status of a specific release:

helm status my-nginx

Uninstalling a Helm Release

To remove the my-nginx release and all associated resources:

helm uninstall my-nginx

Understanding Helm Charts

What is a Helm Chart?

A Helm chart is a packaged application definition that contains Kubernetes resource templates and default configuration values. It allows you to deploy complex applications with a single command while keeping configurations modular and reusable.

Each chart defines:

What Kubernetes resources to deploy (e.g., Deployments, Services, ConfigMaps).
How those resources should be configured using a parameterized values file (values.yaml).
Dependencies and metadata required for installation.

Structure of a Helm Chart

When you create a Helm chart, it follows a specific directory structure:

mychart/
│── charts/           # Directory for chart dependencies (other charts)
│── templates/        # Contains Kubernetes YAML templates
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── _helpers.tpl  # Contains reusable template functions
│── Chart.yaml        # Metadata about the chart (name, version, description)
│── values.yaml       # Default configuration values for the chart
│── README.md         # Documentation about the chart

Each file in this structure serves a specific purpose:

Chart.yaml – Contains metadata such as chart name, version, and description.
values.yaml – Defines default values that can be overridden during installation.
templates/ – Holds Kubernetes manifest templates using Helm’s templating syntax.
charts/ – Stores dependencies (other charts required for deployment).
README.md – Documents how to use the chart.

Example: `Chart.yaml`

The Chart.yaml file provides information about the chart:

apiVersion: v2
name: mychart
description: A sample Helm chart for Kubernetes
type: application
version: 1.0.0
appVersion: 1.16.0
name: The chart's name.
description: A brief description of what the chart does.
version: The chart version (used for versioning updates).
appVersion: The application version the chart deploys.

Example: `values.yaml`

The values.yaml file defines default configuration values:

replicaCount: 2

image:
  repository: nginx
  tag: latest
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80

These values can be overridden when installing the chart using the --set flag or a custom values file.

Example: `templates/deployment.yaml`

A sample Kubernetes Deployment template using Helm's templating syntax:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-nginx
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: {{ .Values.service.port }}

In this template:

{{ .Release.Name }} dynamically sets the release name.
{{ .Values.replicaCount }} pulls values from values.yaml.
{{ .Values.image.repository }}:{{ .Values.image.tag }} sets the container image dynamically.

Rendering Helm Templates

Before applying a Helm chart, you can preview how the templates will render using:

helm template mychart

Writing Your Own Helm Chart

Now that we understand Helm charts and their structure, let’s walk through the process of creating a custom Helm chart from scratch.

Step 1: Create a New Helm Chart

To generate a new Helm chart, use the following command:

helm create mychart

This command creates a new directory mychart/ with the standard Helm chart structure.

Step 2: Modify values.yaml

Open values.yaml and update it with custom values. Let’s modify it to deploy an NGINX web server with a LoadBalancer service:

replicaCount: 3

image:
  repository: nginx
  tag: latest
  pullPolicy: IfNotPresent

service:
  type: LoadBalancer
  port: 80

replicaCount: Defines how many replicas the deployment will create.
image: Configures the container image.
service: Sets the service type and port.

Step 3: Customize Deployment Template

Edit templates/deployment.yaml to use Helm’s templating syntax:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-nginx
  labels:
    app: nginx
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: {{ .Values.service.port }}

{{ .Release.Name }} dynamically assigns the release name.
{{ .Values.replicaCount }} references values from values.yaml.
{{ .Values.image.repository }}:{{ .Values.image.tag }} configures the image dynamically.

Step 4: Customize the Service Template

Edit templates/service.yaml to configure the service:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}-nginx
spec:
  type: {{ .Values.service.type }}
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: {{ .Values.service.port }}
      targetPort: {{ .Values.service.port }}

Step 5: Package the Helm Chart

Once you've modified the necessary files, package the chart:

helm package mychart

This creates a .tgz archive of the chart, making it ready for distribution.

Step 6: Install the Chart

Deploy the chart to your Kubernetes cluster:

helm install my-nginx ./mychart

This:

Parses templates.
Replaces placeholders with values from values.yaml.
Applies the resources to Kubernetes.

Step 7: Verify the Deployment

Check the deployed resources:

helm list
kubectl get pods
kubectl get svc

Step 8: Uninstall the Chart

To remove the deployment, use:

helm uninstall my-nginx

Deploying Applications with Helm

Once you've created or downloaded a Helm chart, you can use Helm to deploy and manage applications in your Kubernetes cluster. This section will walk through the deployment process, including installation, upgrades, rollbacks, and uninstallation.

Step 1: Installing a Helm Chart

To deploy an application using Helm, use the helm install command:

helm install my-nginx ./mychart

my-nginx is the release name (a unique identifier for this deployment).
./mychart is the path to the Helm chart.

If you are installing a chart from a repository, such as Bitnami, use:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-nginx bitnami/nginx

This command:

Pulls the nginx chart from the Bitnami repository.
Deploys NGINX to the Kubernetes cluster.
Creates a Helm release named my-nginx.

Step 2: Verifying the Deployment

Once the chart is installed, verify that the release is active:

helm list

This will output something like:

NAME        NAMESPACE   REVISION    UPDATED                  STATUS      CHART        APP VERSION
my-nginx    default     1           2024-02-16 10:00:00     deployed    nginx-1.2.3  1.21.6

You can check the detailed status of a release:

helm status my-nginx

To view the created Kubernetes resources:

kubectl get pods
kubectl get svc

Step 3: Customizing Helm Releases

Helm allows you to override default values using the --set flag or a custom values file.

Using the `--set` Flag

You can override individual values like this:

helm install my-nginx bitnami/nginx --set replicaCount=3

Using a Custom values.yaml File

To provide multiple custom values, create a my-values.yaml file:

replicaCount: 3
service:
  type: LoadBalancer
  port: 8080

Then, deploy the chart with:

helm install my-nginx bitnami/nginx -f my-values.yaml

Step 4: Upgrading a Helm Release

If you need to modify a running deployment, use the helm upgrade command:

helm upgrade my-nginx bitnami/nginx --set replicaCount=5

To upgrade using a modified values file:

helm upgrade my-nginx bitnami/nginx -f my-values.yaml

This updates the deployment while keeping existing resources intact.

Step 5: Rolling Back to a Previous Version

Helm maintains a history of releases, allowing you to roll back if needed.

List the release history:

helm history my-nginx

Roll back to a specific revision:

helm rollback my-nginx 1

Step 6: Uninstalling a Helm Release

To remove a Helm deployment and all its associated resources, run:

helm uninstall my-nginx

To confirm deletion:

helm list
kubectl get all

Helm Best Practices

Using Helm effectively requires following best practices to ensure maintainability, security, and scalability of deployments. This section outlines key strategies for optimizing Helm usage in production environments.

1. Organizing Values in `values.yaml` for Clarity

A well-structured values.yaml file improves readability and maintainability.

✅ Good Example: Structured and Documented

replicaCount: 3  # Number of replicas for high availability

image:
  repository: nginx
  tag: latest
  pullPolicy: IfNotPresent  # Pull policy to optimize image fetching

service:
  type: LoadBalancer
  port: 80  # Publicly exposed service port

resources:
  limits:
    cpu: 500m
    memory: 256Mi
  requests:
    cpu: 250m
    memory: 128Mi

❌ Bad Example: Unstructured and Unclear

replicaCount: 3
image: nginx:latest
serviceType: LoadBalancer
port: 80
cpu: 500m
memory: 256Mi

No clear nesting.
Missing descriptions for future maintainers.
Harder to override values at a granular level.

2. Using helm dependency for Managing Dependencies

If your chart depends on other charts (e.g., a database), declare them in Chart.yaml:

dependencies:
  - name: postgresql
    version: "12.1.3"
    repository: "https://charts.bitnami.com/bitnami"

Then, update dependencies before installing:

helm dependency update

This ensures that all required subcharts are installed and properly versioned.

3. Leveraging helm secrets for Sensitive Values

Avoid storing credentials in values.yaml. Instead, use Helm Secrets to encrypt sensitive values.

Install the Helm Secrets plugin:

helm plugin install https://github.com/zachomedia/helm-secrets

Encrypt sensitive values using SOPS:

sops --encrypt --in-place my-values.yaml

Install a chart using encrypted values:

helm install my-app ./mychart -f my-values.yaml

This ensures secrets are not stored in plaintext inside version control.

4. Automating Helm Deployments in CI/CD Pipelines

Integrate Helm with CI/CD tools like GitHub Actions, GitLab CI/CD, or ArgoCD to automate deployments.

Example GitHub Actions Workflow for Helm

name: Deploy Helm Chart

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Install Helm
        run: |
          curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

      - name: Deploy to Kubernetes
        run: |
          helm upgrade --install my-app ./mychart --namespace prod

This automates deployments whenever code is pushed to the main branch.

5. Keeping Charts Versioned and Documented

Use semantic versioning in Chart.yaml (version: 1.2.0).
Document all available values in README.md. Maintain a CHANGELOG.md to track modifications.

#6. Managing Multiple Environments (Dev, Staging, Prod)

Helm allows environment-specific values with separate values files:

helm install my-app ./mychart -f values-dev.yaml
helm install my-app ./mychart -f values-prod.yaml

This ensures different configurations for testing and production.

7. Helm Security Considerations

Avoid running Helm with cluster-wide privileges.
Restrict Helm Release Names to prevent namespace conflicts.
Use RBAC policies to limit Helm access. Regularly update Helm and chart dependencies to patch vulnerabilities.

Summary

Organize values.yaml clearly for maintainability.
Use helm dependency to manage subcharts.
Secure sensitive values with helm secrets and encryption.
Automate Helm deployments using CI/CD.
Maintain versioning, documentation, and separate environments.
Follow security best practices to protect Kubernetes resources.
In the next section, we’ll discuss Helm’s role in large-scale production deployments and how to integrate it with GitOps tools like ArgoCD and Flux.

Helm in Production: Managing Complexity at Scale

As organizations scale their Kubernetes deployments, managing Helm charts effectively in production becomes crucial. This section explores how Helm integrates with GitOps tools, supports multi-environment management, and follows best practices for high availability and security.

1. Using GitOps with Helm (ArgoCD & Flux)

GitOps enables declarative infrastructure management, where Helm charts are stored in Git repositories and automatically deployed using tools like ArgoCD and Flux.

Deploying Helm Charts with ArgoCD

ArgoCD monitors a Git repository and applies changes automatically.

Install ArgoCD:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Deploy a Helm Chart with ArgoCD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-helm-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/helm-charts.git
    targetRevision: main
    path: mychart
    helm:
      valueFiles:
        - values-prod.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: prod

Apply the application manifest:

kubectl apply -f my-helm-app.yaml

ArgoCD will now continuously sync the Helm chart with the Kubernetes cluster.

Using FluxCD for Helm Deployments

FluxCD can also automate Helm deployments:

flux create source git my-helm-repo \
  --url=https://github.com/my-org/helm-charts.git \
  --branch=main

flux create helmrelease my-app \
  --source=GitRepository/my-helm-repo \
  --chart=mychart \
  --namespace=prod

GitOps ensures:

Automated rollouts & rollbacks when changes are pushed to Git.
Version-controlled infrastructure for reproducibility.
Improved collaboration by managing Helm charts as code.

2. Managing Multi-Cluster Deployments

For enterprises running multiple Kubernetes clusters (e.g., dev, staging, prod), Helm enables consistent deployments across environments.

Option 1: Context Switching with kubectl

kubectl config use-context dev-cluster
helm install my-app ./mychart --namespace dev

kubectl config use-context prod-cluster
helm install my-app ./mychart --namespace prod

Option 2: Using Helmfile for Multi-Cluster Deployments

Helmfile allows managing multiple Helm releases in a declarative format.

Example helmfile.yaml:

releases:
  - name: my-app-dev
    namespace: dev
    chart: ./mychart
    values:
      - values-dev.yaml

  - name: my-app-prod
    namespace: prod
    chart: ./mychart
    values:
      - values-prod.yaml

Deploy all environments at once:

helmfile apply

3. Ensuring High Availability and Reliability

Use Helm Hooks: Automate pre-install and post-install tasks.

annotations:
  "helm.sh/hook": pre-install

Enable Readiness and Liveness Probes to ensure application health:

readinessProbe:
  httpGet:
    path: /
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10

Use Rolling Updates with strategy to prevent downtime:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1
    maxSurge: 1

4. Helm Security Best Practices for Production

Restrict Helm Permissions using Role-Based Access Control (RBAC):

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: prod
  name: helm-user
rules:
  - apiGroups: ["*"]
    resources: ["deployments", "services"]
    verbs: ["get", "list", "create", "update", "delete"]

Avoid Storing Secrets in values.yaml:

Use Kubernetes Secrets and refer to them in Helm templates. En- crypt secrets with SOPS or use External Secrets Operator.

Implement Image Scanning:

Use tools like Trivy or Anchore to scan Helm charts and container images.

Regularly Update Helm and Charts:

Ensure Helm CLI and chart dependencies are up to date.
Use helm dependency update to pull the latest versions.

5. Monitoring and Logging Helm Deployments

Track Helm Releases:

helm list --all-namespaces

Monitor Deployments with Prometheus & Grafana:

Install Prometheus using Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus

Integrate with Grafana for dashboard visualization.

Use Helm Logs to Debug Issues:

helm get manifest my-app
helm get values my-app
helm get notes my-app

Summary

GitOps tools (ArgoCD, Flux) enable automated Helm deployments.
Multi-cluster management can be streamlined with Helmfile or Helm contexts.
High availability practices ensure smooth rolling updates and failovers.
Security best practices include using RBAC, encrypted secrets, and image scanning.
Monitoring tools like Prometheus and Grafana help track Helm deployments.

9. Conclusion and Next Steps

Helm simplifies Kubernetes application deployment, making it easier to manage complex workloads with reusable, version-controlled charts. By leveraging Helm, teams can standardize configurations, automate deployments, and integrate with GitOps workflows to achieve reliable and scalable Kubernetes operations.

Key Takeaways

Helm is the Kubernetes Package Manager – It streamlines application deployments by packaging Kubernetes resources into reusable Helm charts.
Charts Provide Flexibility – Using values.yaml, teams can easily override configurations without modifying templates.
Helm Supports Versioning & Rollbacks – The ability to upgrade and roll back releases ensures stability and rapid recovery.
Automation & CI/CD Integration – Helm works seamlessly with GitOps tools like ArgoCD and FluxCD to automate deployments.
Security & Best Practices Matter – Implement RBAC, use secrets management, and ensure chart dependencies are up to date to maintain a secure and efficient Helm workflow.
Monitoring & Debugging Are Essential – Use Prometheus, Grafana, and Helm’s built-in commands (helm list, helm get) to track deployments and troubleshoot issues.

Next Steps: Continue Learning Helm

Now that you understand Helm’s capabilities, here are some next steps to deepen your knowledge and practical experience:

Explore Official Helm Documentation
📌 Helm Docs
Deploy Real-World Applications with Helm
- Try deploying WordPress, PostgreSQL, or Redis with Helm charts from Artifact Hub.
- Example:
```
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-wordpress bitnami/wordpress
```
Experiment with Custom Helm Charts
- Modify an existing chart or build one from scratch.
- Deploy it to different environments using separate values.yaml files.
Integrate Helm with a CI/CD Pipeline
- Set up GitHub Actions, GitLab CI/CD, or Jenkins to automate Helm deployments.
Learn Advanced Helm Features
- Helm Hooks: Automate tasks before/after deployments.
- Helm Subcharts: Manage dependencies efficiently.
- Helm Secrets: Encrypt sensitive configurations.
Follow Helm & Kubernetes Communities
- Join the CNCF Slack (#helm-users channel).
- Follow Kubernetes and Helm GitHub discussions for the latest updates.

Final Thoughts

Helm is an essential tool for Kubernetes administrators and DevOps teams looking to optimize deployment workflows. Whether you are deploying simple microservices or complex cloud-native applications, Helm provides the flexibility, automation, and reliability needed to scale efficiently.

Start experimenting with Helm today and take your Kubernetes skills to the next level!

Additional Resources

Helm Charts Repository: Artifact Hub
Kubernetes Documentation: Kubernetes.io
ArgoCD for Helm: ArgoCD Docs
FluxCD for Helm: FluxCD Docs
Helm Security Best Practices: Helm Security Guide

Crash Course on Developing AI Applications with LangChain

Alex Merced — Sat, 01 Feb 2025 09:00:00 GMT

Free Resources

Introduction

Large Language Models (LLMs) have revolutionized the way developers build AI-powered applications, from chatbots to intelligent search systems. However, managing LLM interactions effectively—structuring prompts, handling memory, and integrating external tools—can be complex. This is where LangChain comes in.

LangChain is an open-source framework designed to simplify working with LLMs, enabling developers to create powerful AI applications with ease. By providing a modular approach, LangChain allows you to compose prompt templates, chains, memory, and agents to build flexible and scalable solutions.

In this guide, we'll introduce you to LangChain and its companion libraries, including:

langchain_community: A collection of core integrations and utilities.
langchain_openai: A dedicated library for working with OpenAI models.

We'll walk you through key LangChain concepts, installation steps, and practical code examples to help you get started. Whether you're looking to build chatbots, AI-powered search engines, or decision-making agents, this guide will give you the foundation you need to start developing with LangChain.

What is LangChain?

LangChain is an open-source framework that simplifies building applications powered by Large Language Models (LLMs). Instead of manually handling prompts, API calls, and responses, LangChain provides a structured way to chain together different components such as prompts, memory, and external tools.

Why Use LangChain?

Without LangChain, interacting with an LLM typically involves:

Formatting a prompt manually.
Sending the request to an API (e.g., OpenAI, Cohere).
Parsing the response and deciding the next action.

LangChain automates and streamlines these steps, making it easier to build complex AI applications with minimal effort.

Key Use Cases

LangChain is widely used for:

Chatbots & Virtual Assistants – Retaining conversation context and improving responses.
Retrieval-Augmented Generation (RAG) – Enhancing LLM responses by fetching external data sources.
Data Processing & Summarization – Analyzing and summarizing large documents.
AI Agents – Creating autonomous agents that interact with external APIs and databases.

By leveraging LangChain’s modular architecture, you can integrate various models, tools, and memory mechanisms to build dynamic AI-driven applications.

Core Concepts in LangChain

LangChain is built around a modular architecture that allows developers to compose different components into a pipeline. Here are some of the key concepts you need to understand when working with LangChain:

1. Prompt Templates

Prompt templates help structure the input given to an LLM. Instead of writing static prompts, you can create dynamic templates that format user inputs into well-structured queries.

Example:

from langchain.prompts import PromptTemplate

template = PromptTemplate(
    input_variables=["topic"],
    template="Explain {topic} in simple terms."
)

formatted_prompt = template.format(topic="LangChain")
print(formatted_prompt)

This ensures that every input follows a structured format before being passed to the model.

2. LLMs and Model Wrappers

LangChain provides an easy way to interface with different LLM providers like OpenAI, Hugging Face, and more.

Example:

from langchain_openai import OpenAI

llm = OpenAI(api_key="your_api_key")
response = llm("What is LangChain?")
print(response)

This allows you to seamlessly query the LLM without worrying about API details.

3. Chains

Chains allow you to combine multiple components (e.g., a prompt template and an LLM) into a single workflow.

Example:

from langchain.chains import LLMChain

llm_chain = LLMChain(llm=llm, prompt=template)
response = llm_chain.run("machine learning")
print(response)

Here, the prompt is formatted and automatically passed to the LLM, reducing boilerplate code.

4. Memory

Memory allows your application to retain context between interactions, which is crucial for chatbots and multi-turn conversations.

Example:

from langchain.memory import ConversationBufferMemory

memory = ConversationBufferMemory()
memory.save_context({"input": "Hello"}, {"output": "Hi, how can I help you?"})
print(memory.load_memory_variables({}))

With memory, LangChain can track past interactions and use them to generate more coherent responses.

5. Agents and Tools

Agents allow an LLM to make decisions dynamically. Instead of following a predefined sequence, an agent determines which tool to call based on the user’s query.

Example:

from langchain.agents import initialize_agent, AgentType
from langchain.tools import Tool

def add_numbers(a, b):
    return a + b

tool = Tool(
    name="Calculator",
    func=add_numbers,
    description="Adds two numbers."
)

agent = initialize_agent(
    tools=[tool],
    llm=llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION
)

response = agent.run("What is 3 + 5?")
print(response)

This enables an LLM to call functions, fetch data, or interact with APIs to generate more intelligent responses.

By understanding these core concepts, you can start building more structured and powerful AI applications with LangChain. In the next section, we’ll set up LangChain and its companion libraries to start developing real-world applications.

Installing LangChain and Companion Libraries

Before we start building with LangChain, we need to install the necessary packages. LangChain is modular, meaning that different functionalities are split across separate libraries. The main ones you'll need are:

langchain – The core LangChain library.
langchain_community – A collection of integrations for third-party tools and services.
langchain_openai – A dedicated package for working with OpenAI models.
openai – The OpenAI Python SDK for API access.

1. Installing LangChain and Dependencies

You can install the required libraries using pip:

pip install langchain langchain_community langchain_openai openai

This command will install the core LangChain framework along with the OpenAI integration.

2. Setting Up an OpenAI API Key

If you plan to use OpenAI models, you’ll need an API key. Follow these steps:

Sign up at OpenAI.
Navigate to your API settings and generate an API key.
Store your API key securely.

You can set your API key in an environment variable:

export OPENAI_API_KEY="your_api_key_here"

Or pass it directly in your code:

import os
os.environ["OPENAI_API_KEY"] = "your_api_key_here"

3. Verifying the Installation

To test if everything is installed correctly, run the following Python script:

from langchain_openai import OpenAI

llm = OpenAI(api_key="your_api_key_here")
response = llm("Say hello in French.")
print(response)

If you see the response "Bonjour!", then your setup is working properly.

4. Understanding the Role of Companion Libraries

langchain_community: Contains integrations for databases, vector stores, and APIs.
langchain_openai: A streamlined package for interacting with OpenAI's models.
Other integrations: LangChain supports many LLM providers (Cohere, Hugging Face, etc.), which can be installed separately.

With LangChain and its dependencies installed, you're ready to start building AI-powered applications. In the next section, we'll explore how to use LangChain with OpenAI models and create structured workflows.

Setting Up and Using LangChain

Now that we have LangChain installed, let's explore how to use it for interacting with LLMs, structuring prompts, and building simple AI workflows.

1. Connecting to an OpenAI Model

The first step in using LangChain is to connect to an LLM. We'll start by using OpenAI's models.

Example: Basic Query to an OpenAI Model

from langchain_openai import OpenAI

llm = OpenAI(api_key="your_api_key_here")

response = llm.invoke("What is LangChain?")
print(response)

This sends a query to OpenAI and prints the response. The invoke method is the recommended way to interact with LLMs in LangChain.

2. Working with Prompt Templates

A prompt template ensures that user input is formatted consistently before being sent to an LLM. This is useful when you need structured responses.

Example: Creating and Using a Prompt Template

from langchain.prompts import PromptTemplate

template = PromptTemplate(
    input_variables=["topic"],
    template="Explain {topic} in simple terms."
)

formatted_prompt = template.format(topic="machine learning")
print(formatted_prompt)

This generates a properly structured prompt: "Explain machine learning in simple terms."

You can pass this formatted prompt to an LLM for processing.

3. Building a Basic Chain

A chain connects multiple components, such as prompts and LLMs, to automate workflows.

Example: Using a Chain to Generate Responses

from langchain.chains import LLMChain

llm_chain = LLMChain(llm=llm, prompt=template)
response = llm_chain.run("data science")
print(response)

Here, LangChain automatically formats the prompt and sends it to the LLM, reducing manual effort.

4. Using Memory to Maintain Context

By default, LLMs don’t remember past interactions. LangChain provides memory components to store and retrieve conversation history.

Example: Storing Conversation History

from langchain.memory import ConversationBufferMemory

memory = ConversationBufferMemory()

# Simulating a conversation
memory.save_context({"input": "Hello"}, {"output": "Hi, how can I help you?"})
memory.save_context({"input": "What is LangChain?"}, {"output": "LangChain is a framework for working with LLMs."})

# Retrieving stored interactions
print(memory.load_memory_variables({}))

This ensures that previous interactions can be referenced in future queries.

5. Implementing an Agent with Tools

An agent allows LLMs to dynamically decide which tool to use for a given query. For example, we can create an agent that uses a calculator tool.

Example: Creating an Agent to Perform Calculations

from langchain.agents import initialize_agent, AgentType
from langchain.tools import Tool

# Defining a simple addition function
def add_numbers(a, b):
    return a + b

tool = Tool(
    name="Calculator",
    func=add_numbers,
    description="Adds two numbers."
)

# Creating an agent with the tool
agent = initialize_agent(
    tools=[tool],
    llm=llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION
)

# Running the agent
response = agent.run("What is 5 + 7?")
print(response)

This enables the LLM to recognize when to use the calculator tool instead of responding based purely on its pre-trained knowledge.

What’s Next?

Now that we've covered basic LangChain functionalities, you can start experimenting with more advanced features like:

Retrieval-Augmented Generation (RAG) – Enhancing LLMs with external knowledge sources.
Vector Databases – Storing and retrieving information efficiently.
Custom Tools and APIs – Expanding agents to interact with real-world data.

In the next section, we'll discuss best practices for using LangChain efficiently and how to scale applications for production use.

Best Practices and Next Steps

Now that you understand the basics of LangChain—connecting to LLMs, structuring prompts, using chains, memory, and agents—let’s discuss some best practices for building efficient and scalable applications.

1. Optimize Prompt Engineering

Use clear and structured prompt templates to get better responses from LLMs.
Experiment with few-shot learning by providing example inputs and outputs.
Keep prompts concise to reduce token usage and improve performance.

Example: Few-Shot Prompting

from langchain.prompts import PromptTemplate

template = PromptTemplate(
    input_variables=["word"],
    template="Convert the following word into plural form: {word}\n\nExample:\n- dog -> dogs\n- cat -> cats\n- book -> ?"
)

print(template.format(word="tree"))

Providing examples improves the model's accuracy.

2. Use Memory Efficiently

Only use conversation memory when necessary (e.g., chatbots).

Choose the right memory type:

ConversationBufferMemory – Stores all conversation history.
ConversationSummaryMemory – Summarizes past interactions.
ConversationKGMemory – Extracts key facts from a conversation.

Example: Using Summary Memory

from langchain.memory import ConversationSummaryMemory
from langchain_openai import OpenAI

llm = OpenAI(api_key="your_api_key")
memory = ConversationSummaryMemory(llm=llm)

memory.save_context({"input": "I love pizza."}, {"output": "Pizza is a great choice!"})
summary = memory.load_memory_variables({})
print(summary)

This helps reduce storage while maintaining context.

3. Handle API Costs and Rate Limits

Use token-efficient prompts to reduce API costs. Implement batch processing for multiple queries. Monitor API usage with OpenAI’s rate limits in mind.

Example: Monitoring Token Usage

from langchain_openai import OpenAI

llm = OpenAI(api_key="your_api_key", model="gpt-4", max_tokens=100)
response = llm("Summarize the history of AI in 50 words.")
print(response)

Setting max_tokens prevents excessive token consumption.

4. Enhance LLMs with External Knowledge (RAG)

Retrieval-Augmented Generation (RAG) improves LLM responses by fetching external data instead of relying solely on pre-trained knowledge.

Use vector databases like Pinecone or FAISS for document search.
Fetch real-time data from APIs.

Example: Querying an External Document

from langchain.vectorstores import FAISS
from langchain.embeddings.openai import OpenAIEmbeddings

# Load and embed documents
embeddings = OpenAIEmbeddings(api_key="your_api_key")
vectorstore = FAISS.load_local("faiss_index", embeddings)

# Query the knowledge base
docs = vectorstore.similarity_search("What is LangChain?", k=2)
print(docs)

This retrieves relevant documents to supplement the LLM’s response.

5. Scale Applications for Production

When moving from prototyping to production, consider:

Caching responses to avoid redundant API calls.
Logging interactions for debugging and improvement.
Implementing user authentication for secured access.

Example: Implementing Response Caching

from langchain.cache import InMemoryCache
from langchain.chains import LLMChain

llm_chain = LLMChain(llm=llm, prompt=template)
llm_chain.cache = InMemoryCache()  # Enable caching

response1 = llm_chain.run("machine learning")
response2 = llm_chain.run("machine learning")  # Cached response
print(response2)

Caching reduces API calls, improving performance and cost-efficiency.

Conclusion

LangChain provides a powerful framework for building AI applications that leverage Large Language Models (LLMs). By combining prompt engineering, chains, memory, and agents, LangChain simplifies the development process, making it easier to create chatbots, AI assistants, and retrieval-augmented generation (RAG) applications.

In this guide, we covered:

What LangChain is and why it’s useful.
Core concepts like prompt templates, chains, memory, and agents.
How to install and set up LangChain along with langchain_openai and langchain_community.
Practical code examples for using LangChain with OpenAI models.
Best practices for optimizing prompts, managing memory, and reducing API costs.
How to scale LangChain applications for production using caching and external knowledge retrieval.

By applying these concepts, you can start building custom AI-powered solutions with real-world impact.

Where to Go from Here?

If you're ready to take the next step, consider:

Building a LangChain Project – Try creating a chatbot, document summarizer, or an AI-driven search engine.
Exploring Vector Databases – Learn how to integrate Pinecone, FAISS, or ChromaDB for RAG applications.
Joining the Community – Engage with other developers on LangChain's GitHub or Discord.

LangChain is continuously evolving, and staying updated with the latest features will help you build more advanced and efficient AI applications. Start experimenting and bring your AI ideas to life!

The Data Lakehouse - The Benefits and Enhancing Implementation

Alex Merced — Fri, 31 Jan 2025 09:00:00 GMT

Free Resources

Introduction

The data lakehouse has been a significant topic in data architecture over the past several years. However, like any high-value trend, it’s easy to get caught up in the hype and lose sight of the real reasons for adopting this new paradigm.

In this article, I aim to clarify the key benefits of a lakehouse, highlight the challenges organizations face in implementing one, and explore practical solutions to overcome those challenges.

The Problems We Are Trying to Solve For

Traditionally, running analytics directly on operational databases (OLTP systems) is neither performant nor efficient, as it creates resource contention with transactional workloads that power enterprise operations. The standard solution has been to offload this data into a data warehouse, which optimizes storage for analytics, manages data efficiently, and provides a processing layer for analytical queries.

However, not all data is structured or fits neatly into a data warehouse. Additionally, storing all structured data in a data warehouse can be cost-prohibitive. As a result, an intermediate layer—a data lake—is often introduced, where copies of data are stored for ad hoc analysis on distributed storage systems like Amazon S3, ADLS, MinIO, NetApp StorageGRID, Vast Data, Pure Storage, Nutanix, and others.

In large enterprises, different business units often choose different data warehouses, leading to multiple copies of the same data, inconsistently modeled across departments. This fragmentation introduces several challenges:

1. Consistency

With multiple copies, business metrics can have different definitions and values depending on which department’s data model you reference, leading to discrepancies in decision-making.

2. Time to Insight

As data volumes grow and the demand for real-time or near real-time insights increases, excessive data movement becomes a bottleneck. Even if individual transactions are fast, the cumulative impact of copying and processing delays data accessibility.

3. Centralization Bottlenecks

To improve consistency, some organizations centralize modeling in an enterprise-wide data warehouse with department-specific data marts. However, this centralization can create bottlenecks, slowing down access to insights.

4. Cost

Every step of data movement incurs costs:

Compute resources for processing,
Storage costs for redundant copies, and
BI tool expenses from multiple teams generating similar data extracts across different tools.

5. Governance

Not all enterprise data resides in a data warehouse. There will always be a long tail of data in external systems, such as:

Partner-shared data,
Data marketplaces, or
Regulatory-restricted environments.

Managing access to a holistic data picture while maintaining governance and security across distributed sources is a significant challenge.

This is where the data lakehouse emerges as a solution.

The Data Lakehouse Solution

Data warehouses provide essential data management capabilities and ACID guarantees, ensuring consistency and reliability in analytics. However, these features have traditionally been absent from data lakes, as data lakes are not inherently data platforms but repositories of raw data stored on open storage.

If we bring data management and ACID transactions to the data lake, organizations can work with a single canonical copy directly within the lake, eliminating the need to replicate data across multiple data warehouses. This transformation turns the data lake into a data warehouse—hence the term data lakehouse.

This is achieved by adopting:

Lakehouse Table Formats like Apache Iceberg, Apache Hudi, Delta Lake, or Apache Paimon, enabling Parquet files to act as structured, ACID-compliant tables optimized for analytics.
Lakehouse Catalogs like Apache Polaris, Nessie, Apache Gravitino, Lakekeeper, and Unity, which provide metadata tracking for seamless data discovery and access.
Managed Catalog Services (e.g., Dremio), which automate data optimization and governance, reducing unnecessary data movement.

Key Benefits of a Lakehouse Approach

✅ Lower costs by reducing data replication and processing overhead.
✅ Improved consistency by maintaining a single source of truth.
✅ Faster time to insight with direct access to analytics-ready data.

Challenges That Remain

Despite its advantages, a lakehouse alone does not completely solve all challenges:

Migration Delays – Moving existing data takes time, delaying the full benefits of a lakehouse.
Distributed Data Sources – Not all data resides in the lakehouse; external data remains a challenge.
BI Tool Extracts – Users may still create redundant isolated extracts, increasing costs.

This is where the Dremio Lakehouse Platform fills the gap.

The Dremio Solution

Dremio is a lakehouse platform that integrates four key capabilities into a holistic data integration solution, addressing the remaining lakehouse challenges.

1. High-Performance Federated Query Engine

Best-in-class raw query performance.
Federates queries across lakehouse catalogs, data lakes, databases, and warehouses.
Provides a centralized experience across disparate data sources.

2. Semantic Layer

Enables virtual data marts without data duplication.
Built-in wiki and search for dataset documentation.
Standardizes business metrics and datasets across all tools.

3. Query Acceleration

Reflections replace traditional materialized views and BI cubes.
Raw Reflections (precomputed query results).
Aggregate Reflections (optimized aggregations for fast analytics).
Automatic query acceleration, with no effort required from analysts.

4. Integrated Lakehouse Catalog

Tracks and manages Apache Iceberg tables.
Automates maintenance and cleanup of data.
Provides centralized, portable governance across all queries.

The Dremio Advantage

✅ Instant Lakehouse Benefits – Get the advantages immediately, even before full migration.
✅ Improved Consistency – Ensure a unified definition of business metrics.
✅ High-Performance Analytics – Federated queries + Reflections accelerate workloads.
✅ Automated Management – No manual cleanup of lakehouse tables needed.
✅ Centralized Governance – Unified access control across all tools and sources.

Conclusion

The data lakehouse represents a transformative shift in data architecture, solving long-standing challenges around data consistency, cost, and accessibility.

However, simply adopting a lakehouse format isn’t enough. Organizations need a lakehouse solution that integrates data management, acceleration, and governance to fully unlock the benefits.

Dremio provides that missing piece with:
✅ Federated query capabilities
✅ A built-in semantic layer
✅ Automated query acceleration
✅ A fully managed lakehouse catalog

With Dremio, organizations don’t just implement a lakehouse—they enhance it, unlocking its full potential for faster insights, better decision-making, and long-term cost savings.

Free Resources

2025 Comprehensive Guide to Apache Iceberg

Alex Merced — Mon, 20 Jan 2025 09:00:00 GMT

Apache Iceberg had a monumental 2024, with significant announcements and advancements from major players like Dremio, Snowflake, Databricks, AWS, and other leading data platforms. The Iceberg ecosystem is evolving rapidly, making it essential for professionals to stay up-to-date with the latest innovations. To help navigate this ever-changing space, I’m introducing an annual guide dedicated to Apache Iceberg. This guide aims to provide a comprehensive overview of Iceberg, highlight key resources, and offer valuable insights for anyone looking to deepen their knowledge. Whether you’re just starting with Iceberg or are a seasoned user, this guide will serve as your go-to resource for 2025.

Read this article for details on migrating to Apache Iceberg.

What is a Table Format?

A table format, often referred to as an “open table format” or “lakehouse table format,” is a foundational component of the data lakehouse architecture. This architecture is gaining popularity for its ability to address the complexities of modern data management. Table formats transform how data stored in collections of analytics-optimized Parquet files is accessed and managed. Instead of treating these files as standalone units to be opened and read individually, a table format enables them to function like traditional database tables, complete with ACID guarantees.

With a table format, users can interact with data through SQL to create, read, update, and delete records, bringing the functionality of a data warehouse directly to the data lake. This capability allows enterprises to treat their data lake as a unified platform, supporting both data warehousing and data lake use cases. It also enables teams across an organization to work with a single copy of data in their tool of choice — whether for analytics, machine learning, or operational reporting — eliminating redundant data movements, reducing costs, and improving consistency across the enterprise.

Currently, there are four primary table formats driving innovation in this space:

Apache Iceberg: Originating from Netflix, this blog’s focus, Iceberg is known for its flexibility and robust support for big data operations.
Delta Lake: Developed by Databricks, it emphasizes simplicity and seamless integration with their ecosystem.
Apache Hudi: Created by Uber, Hudi focuses on real-time data ingestion and incremental processing.
Apache Paimon: Emerging from the Apache Flink Project, Paimon is designed to optimize streaming and batch processing use cases.

Each of these table formats plays a role in the evolving data lakehouse landscape, enabling organizations to unlock the full potential of their data lakehouse.

How Table Formats Work

At the core of every table format is a metadata layer that transforms collections of files into a table-like structure. This metadata serves as a blueprint for understanding the data, providing essential details such as:

Files included in the table: Identifying the physical Parquet or similar files that make up the dataset.
Partitioning scheme: Detailing how the data is partitioned to optimize query performance.
Schema: Defining the structure of the table, including column names, data types, and constraints.
Snapshot history: Tracking changes over time, such as additions, deletions, and updates to the table, enabling features like time travel and rollback.

This metadata acts as an entry point, allowing tools to treat the underlying files as a cohesive table. Instead of scanning all files in a directory, query engines use the metadata to understand the structure and contents of the table. Additionally, the metadata often includes statistics about partitions and individual files. These statistics enable advanced query optimization techniques, such as pruning or skipping files that are irrelevant to a specific query, significantly improving performance.

While all table formats rely on metadata to bridge the gap between raw files and table functionality, each format structures and optimizes its metadata differently. These differences can influence performance, compatibility, and the features each format provides.

How Apache Iceberg’s Metadata is Structured

Apache Iceberg’s metadata structure is what enables it to transform raw data files into highly performant and queryable tables. This structure consists of several interrelated components, each designed to provide specific details about the table and optimize query performance. Here’s an overview of Iceberg’s key metadata elements:

metadata.json:
- The metadata.json file is the primary entry point for understanding the table.
- This semi-structured JSON object contains information about the table’s schema, partitioning scheme, snapshot history, and other critical details.
Manifest List:
- Each snapshot in Iceberg has a corresponding Avro-based “manifest list.” This list contains rows representing each manifest (a group of files) that makes up the snapshot.
- Each row includes:
  - The file location of the manifest.
  - Partition value information for the files in the manifest.
- This information allows query engines to prune unnecessary manifests and avoid scanning irrelevant partitions, improving query efficiency.
Manifests:
- A manifest lists one or more Parquet files and includes statistics about each file, such as column summaries.
- These statistics allow query engines to determine whether a file contains data relevant to the query, enabling file skipping for improved performance.
Delete Files:
- Delete files track records that have been deleted as part of “merge-on-read” updates. During queries, the engine reconciles these files with the base data, ensuring that deleted records are ignored.
- There is ongoing discussion about transitioning from delete files to a “deletion vector” approach, inspired by Delta Lake, where deletions are tracked using Puffin files. As of this writing, this proposal has not yet been implemented.
Puffin Files:
- Puffin files are a format for tracking binary blobs and other metadata, designed to optimize queries for engines that choose to leverage them.
Partition Stats Files:
- These files summarize statistics at the partition level, enabling even greater optimization for queries that rely on partitioning.

The Evolution of Iceberg’s Specification

Apache Iceberg’s specification is constantly evolving through community contributions and proposals. These innovations benefit the entire ecosystem, as improvements made by one platform are shared across others. For example:

Partition Stats Files originated from work by Dremio to enhance query optimization.
Puffin Files were introduced by the Trino community to improve how Iceberg tracks metadata.

This collaborative approach ensures that Apache Iceberg continues to evolve as a cutting-edge table format for modern data lakehouses.

Read this article on the Apache Iceberg Metadata tables.

The Role of Catalogs in Apache Iceberg

One of the key features of Apache Iceberg is its immutable file structure, which makes snapshot isolation possible. Every time the data or structure of a table changes, a new metadata.json file is generated. This immutability raises an important question: how does a tool know which metadata.json file is the latest one?

This is where Lakehouse Catalogs come into play. A Lakehouse Catalog serves as an abstraction layer that tracks each table’s name and links it to the most recent metadata.json file. When a table’s data or structure is updated, the catalog is also updated to point to the new metadata.json file. This update is the final step in any transaction, ensuring that the change is completed successfully and meets the atomicity requirement of ACID compliance.

Lakehouse Catalogs are distinct from Enterprise Data Catalogs or Metadata Catalogs, such as those provided by companies like Alation and Collibra. While Lakehouse Catalogs focus on managing the technical details of tables and transactions, enterprise data catalogs are designed for end-users. They act as tools to help users discover, understand, and request access to datasets across an organization, enhancing data governance and usability.

Read this article to learn more about Iceberg catalogs.

The Apache Iceberg REST Catalog Spec

As more catalog implementations emerged, each with unique features and APIs, interoperability between tools and catalogs became a significant challenge. This lack of a unified standard created a bottleneck for seamless table management and cross-platform compatibility.

To address this issue and drive innovation, the REST Catalog specification was developed. Rather than requiring all catalog providers to adopt a standardized server-side implementation, the specification introduced a universal REST API interface. This approach ensures that:

Tools and systems can rely on a consistent, client-side library to interact with catalogs.
Catalog providers maintain the flexibility to implement their server-side systems in ways that suit their needs, as long as they adhere to the standard REST endpoints outlined in the specification.

With the REST Catalog specification, interoperability and ease of integration have dramatically improved. This innovation allows developers and enterprises to adopt or build catalogs that align with their technical and business requirements while still being compatible with any tool that supports the REST API interface. This forward-thinking design has strengthened the role of catalogs in modern lakehouse architectures, ensuring that Iceberg tables remain accessible and manageable across diverse platforms.

Soft Deletes vs Hard Deleting Data

When working with table formats like Apache Iceberg, it’s important to understand how data deletion is handled. Unlike traditional databases, where deleted data is immediately removed from the storage layer, Iceberg follows a different approach to maintain snapshot isolation and enable features like time travel.

When you execute a delete query, the data is not physically deleted. Instead:

A new snapshot is created where the deleted data is no longer present.
The original data files remain intact because the old snapshots are still valid and accessible.

This approach allows users to query previous versions of the table using time travel, providing a powerful mechanism for auditing, debugging, and historical analysis.

However, this also means that data marked for deletion continues to occupy storage until it is physically removed. To address this, snapshot expiration procedures are performed during table maintenance using tools like Spark or Dremio. These procedures:

Invalidate old snapshots that are no longer needed.
Remove the associated data files from storage, freeing up space.

Regular maintenance is a critical part of managing Iceberg tables to ensure storage efficiency and maintain optimal performance while leveraging the benefits of its snapshot-based architecture.

Optimizing Iceberg Data

Minimizing Storage

The first step in reducing storage costs is selecting the right compression algorithm for your data. Compression not only reduces the amount of space required to store data but can also improve performance by accelerating data transfer across networks. These compression settings can typically be adjusted at both the table and query engine levels to suit your specific use case.

Improving Performance

Optimizing performance largely depends on how data is distributed across files. This can be achieved through regular maintenance procedures using tools like Spark or Dremio. These optimizations result in two key outcomes:

Compaction:
- Reduces the number of small files and consolidates delete files into fewer, larger files.
- Minimizes the number of I/O operations required during query execution, leading to faster reads.
Clustering/Sorting:
- Reorganizes data to co-locate similar records within the same files based on commonly queried fields.
- Allows query engines to skip more files during a query, as the data being searched for is concentrated in a smaller subset of files.

By leveraging these strategies, Iceberg users can maintain a balance between efficient storage and fast query performance, ensuring their data lakehouse operates at peak efficiency. Regular maintenance is essential for reaping the full benefits of these optimizations.

Read this article for more detail on optimizing Apache Iceberg tables.