Most teams meet the analytics wall the same way. There is a MySQL database happily serving the application, and someone asks a question that needs a year of history, a window function, and a scan across forty million rows. The query runs. Then it keeps running. Then the on-call engineer gets paged because the read replica fell over and checkout latency went with it.
The classic answer is "buy a warehouse." Before you do that, it is worth spending twenty minutes finding out what a lakehouse actually feels like to work with, on your own laptop, with no cloud account and no credit card.
This is that twenty minutes. By the end you will have a working local pipeline that pulls data from a public API with Node.js, lands it in MySQL, and then uses DuckDB to query MySQL directly, write an Apache Iceberg table, update rows in it, and run a time-travel query against its history. Every command and every line of code in this article was run on a laptop before it was published, and the output you see is the real output.
The Short Version
What you are building: a Node.js script pulls order data from a public API into MySQL. A second Node.js script opens DuckDB, attaches the MySQL database, aggregates it, and writes the result to an Apache Iceberg table. Then it inserts, updates, inspects snapshots, and time travels.
The one thing that will trip you up: DuckDB cannot create an Iceberg table in a plain local folder. Reads work from a path. Writes require an attached REST catalog. That is one extra container in docker-compose, and skipping it is why so many DuckDB Iceberg examples fail on the first CREATE TABLE.
What it costs: nothing. Three containers, two npm packages, about 300 lines total.
Who this is for: backend and full-stack engineers who know SQL and Node.js and have heard "lakehouse" in enough meetings to want to actually touch one.
Why DuckDB and Iceberg Belong Together
These two tools solve opposite halves of the same problem, which is why they pair so well.
Apache Iceberg is a table format. It is not a database and not an engine. It is a specification for describing a table that lives as Parquet files in object storage: which files belong to the table right now, what the schema is, what changed in each commit. That metadata is what turns a folder of Parquet into something with ACID transactions, schema evolution, row-level updates, and a queryable history. If you want the background on why that matters, our guide to lakehouse architecture and the comparison of data warehouse vs data lake vs lakehouse cover the architecture behind the format.
DuckDB is an engine. It is an in-process analytical database, the SQLite of OLAP. It has no server, installs as a library, and reads Parquet, CSV, JSON, Postgres, MySQL, and Iceberg through extensions. It runs a vectorised columnar execution engine that will happily out-run a small Spark cluster on anything that fits on one machine, and "one machine" in 2026 means a few hundred gigabytes.
Put together, you get the storage guarantees of a warehouse and the operational footprint of a library. No cluster, no JVM, no orchestration layer, no per-second billing. For a very large number of real analytical workloads, that combination is enough, and it stays enough for longer than most teams expect.
The genuinely important property is that Iceberg is engine-neutral. The table DuckDB writes in this tutorial can be read by Spark, Trino, Snowflake, Databricks, Flink, or ClickHouse without conversion or export. You are not locked into DuckDB by choosing it today. That is the difference between choosing a format and choosing a vendor.
The Trap: DuckDB Cannot Write Iceberg to a Local Folder
This deserves its own section because it is the single most common failure point, and a fair number of blog posts on the internet get it wrong.
DuckDB's Iceberg support has two separate interfaces:
The path interface is read-only. iceberg_scan('/some/path'), iceberg_metadata(), and iceberg_snapshots() all work against a table directory or a metadata JSON file. If someone hands you an Iceberg table, this is how you read it. You cannot write through it.
The catalog interface is read-write. You ATTACH a catalog and then use normal SQL: CREATE TABLE, INSERT, UPDATE, DELETE, MERGE INTO, ALTER TABLE. Iceberg write support landed in DuckDB 1.4.0 (September 2025) and has been extended since, with MERGE INTO and full schema evolution arriving in 1.5.3.
The catch is that "catalog" here means a REST catalog, or AWS Glue, or S3 Tables. There is no filesystem catalog, no Hadoop catalog, no SQLite catalog. If you try the thing that feels obvious:
ATTACH '/tmp/my_warehouse' AS lake (TYPE iceberg);you get this, on DuckDB 1.5.5:
Invalid Configuration Error: AUTHORIZATION_TYPE is 'oauth2', yet no 'secret' was
provided, and no client_id+client_secret were provided.It is not ignoring your path and defaulting to something. It is unconditionally trying to speak the REST protocol, because that is the only catalog protocol it implements. Passing ENDPOINT_TYPE 'filesystem' or 'hadoop' gets you a blunter answer:
Invalid Configuration Error: Unrecognized 'endpoint_type' (filesystem),
accepted options are: glue, s3_tablesThere is an open feature request to support catalog-free local Iceberg writes. It is not implemented.
So: to write Iceberg locally, you run a REST catalog. That is one extra container, and the Apache Iceberg project publishes an official test fixture image for exactly this purpose. It is genuinely a five-line addition to docker-compose, and it is the honest local equivalent of what you would run in production anyway.
If you want a local lakehouse with zero servers, DuckDB's own DuckLake format does that:
ATTACH 'ducklake:/tmp/lake.ducklake' AS lake (DATA_PATH '/tmp/data/')works with no catalog at all. It is not Iceberg, so it does not give you the cross-engine compatibility. DuckDB ships aniceberg_to_ducklake()function to move between them. Iceberg is the right choice when other engines need to read your tables, which is most of the time.
Prerequisites
- Node.js 20 or newer. The ingest script uses the built-in
fetch, so no HTTP library is needed. This was tested on Node 26. - Docker. Three containers: MySQL, MinIO (S3-compatible object storage), and the Iceberg REST catalog.
- No DuckDB install. The
@duckdb/node-apinpm package ships the engine.
Versions used, all current as of August 2026: DuckDB 1.5.5 (via @duckdb/node-api 1.5.5-r.3), MySQL 8.4, apache/iceberg-rest-fixture.
Step 1: The Infrastructure
Create a project folder and a docker-compose.yml:
services:
mysql:
image: mysql:8.4
environment:
MYSQL_ROOT_PASSWORD: rootpw
MYSQL_DATABASE: shop
MYSQL_USER: shop
MYSQL_PASSWORD: shoppw
ports: ["3306:3306"]
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-prootpw"]
interval: 5s
retries: 20
minio:
image: minio/minio
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
ports: ["9000:9000", "9001:9001"]
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
retries: 20
# Creates the bucket Iceberg writes into, then exits.
minio-init:
image: minio/mc
depends_on:
minio: { condition: service_healthy }
entrypoint: >
/bin/sh -c "
mc alias set local http://minio:9000 minioadmin minioadmin &&
mc mb --ignore-existing local/warehouse"
iceberg-rest:
image: apache/iceberg-rest-fixture
depends_on:
minio-init: { condition: service_completed_successfully }
ports: ["8181:8181"]
environment:
AWS_ACCESS_KEY_ID: minioadmin
AWS_SECRET_ACCESS_KEY: minioadmin
AWS_REGION: us-east-1
CATALOG_WAREHOUSE: s3://warehouse/
CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
CATALOG_S3_ENDPOINT: http://minio:9000
CATALOG_S3_PATH__STYLE__ACCESS: "true"Three moving parts, and the division of labour is worth understanding because it is the same division you will have in production:
- MinIO holds the actual data. Iceberg's Parquet files and JSON metadata land in the
warehousebucket. In production this is S3, GCS, or Azure Blob, and nothing else changes. - iceberg-rest holds the *pointer*. It answers "what is the current state of table X" and arbitrates commits so two writers cannot corrupt each other. In production this is AWS Glue, S3 Tables, Lakekeeper, Polaris, or Unity Catalog.
- MySQL is your operational database, standing in for the one you already have.
Bring it up:
docker compose up -dConfirm the catalog is alive before moving on:
curl -s "http://localhost:8181/v1/config?warehouse=s3://warehouse/"You should get a JSON blob listing the REST endpoints it supports. If you get nothing, give it another ten seconds; the JVM inside takes a moment.
These credentials are hardcoded for a throwaway local stack. Real deployments put them in environment variables or a secrets manager and never in a committed compose file.
Step 2: Node.js Dependencies
npm init -y
npm pkg set type=module
npm install @duckdb/node-api mysql2Two packages. @duckdb/node-api is the modern DuckDB Node client, and it bundles the engine, so there is nothing to install separately.
Step 3: Pull Data From an API Into MySQL
We need order-shaped data. dummyjson.com/carts returns 208 shopping carts containing 800 line items, with products, quantities, prices, and discounts. It is free, needs no key, and looks close enough to real eCommerce data that the queries are meaningful.
Create src/ingest.js:
// Pulls order data from a public API and lands it in MySQL.
import mysql from "mysql2/promise";
const API = "https://dummyjson.com/carts?limit=0";
const db = await mysql.createConnection({
host: process.env.MYSQL_HOST ?? "127.0.0.1",
port: Number(process.env.MYSQL_PORT ?? 3306),
user: process.env.MYSQL_USER ?? "shop",
password: process.env.MYSQL_PASSWORD,
database: process.env.MYSQL_DATABASE ?? "shop",
});
await db.query(`
CREATE TABLE IF NOT EXISTS order_lines (
order_id INT NOT NULL,
product_id INT NOT NULL,
product_name VARCHAR(255) NOT NULL,
quantity INT NOT NULL,
unit_price DECIMAL(10,2) NOT NULL,
line_total DECIMAL(12,2) NOT NULL,
discount_pct DECIMAL(5,2) NOT NULL,
PRIMARY KEY (order_id, product_id)
)
`);
const res = await fetch(API);
if (!res.ok) throw new Error(`${API} responded ${res.status}`);
const { carts } = await res.json();
const rows = carts.flatMap((cart) =>
cart.products.map((p) => [
cart.id,
p.id,
p.title,
p.quantity,
p.price,
p.total,
p.discountPercentage,
])
);
// Re-runnable: same order line just overwrites itself instead of erroring.
const [result] = await db.query(
`INSERT INTO order_lines
(order_id, product_id, product_name, quantity, unit_price, line_total, discount_pct)
VALUES ?
ON DUPLICATE KEY UPDATE
quantity = VALUES(quantity),
line_total = VALUES(line_total)`,
[rows]
);
console.log(`loaded ${rows.length} order lines from ${carts.length} orders (affected: ${result.affectedRows})`);
await db.end();Two details that matter more than they look:
The bulk insert is parameterised. mysql2 expands the VALUES ? placeholder from an array of arrays and escapes every value. Building that SQL by string concatenation is how you write a SQL injection bug into a data pipeline, and pipelines are a soft target precisely because nobody thinks of them as user-facing.
The upsert makes it re-runnable. Pipelines get re-run. A load step that explodes on second execution is a load step you will be babysitting at 3am.
Run it:
MYSQL_PASSWORD=shoppw node src/ingest.jsloaded 800 order lines from 208 orders (affected: 810)Step 4: DuckDB Reads MySQL and Writes Iceberg
This is the interesting part. Create src/analyze.js:
// Reads the operational MySQL table with DuckDB, writes an aggregate into
// Iceberg, then queries the Iceberg table back.
import { DuckDBInstance } from "@duckdb/node-api";
const instance = await DuckDBInstance.create(":memory:");
const db = await instance.connect();
const show = async (label, sql) => {
const reader = await db.runAndReadAll(sql);
console.log(`\n--- ${label} ---`);
console.table(reader.getRowObjectsJson());
};
await db.run(`
INSTALL mysql; LOAD mysql;
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs; LOAD httpfs;
`);
// MinIO stands in for S3 - this is where Iceberg's Parquet data files land.
await db.run(`
CREATE SECRET minio (
TYPE s3,
KEY_ID 'minioadmin', SECRET 'minioadmin',
ENDPOINT '127.0.0.1:9000', URL_STYLE 'path', USE_SSL false
)
`);
await db.run(`
ATTACH 'host=127.0.0.1 port=3306 user=shop password=shoppw database=shop'
AS shop (TYPE mysql, READ_ONLY)
`);
await db.run(`
ATTACH 'warehouse' AS lake (
TYPE iceberg, ENDPOINT 'http://127.0.0.1:8181', AUTHORIZATION_TYPE 'none'
)
`);A few things are worth pausing on here.
getRowObjectsJson() rather than getRowObjects(): DuckDB's BIGINT maps to JavaScript bigint, and JSON.stringify throws on bigint. The ...Json() variants stringify numeric types for you. This will bite you within five minutes of using the client otherwise.
READ_ONLY on the MySQL attach: analytics has no business writing to the operational database. Making that explicit at the connection level is one word and removes a whole class of accident.
ATTACH 'warehouse' and not ATTACH 's3://warehouse/': this one is subtle and will cost you an hour. If the attach string looks like a path, DuckDB attaches the catalog read-only, and your first CREATE TABLE fails with Cannot execute statement of type "CREATE" on database "lake" which is attached in read-only mode!. Pass the bare warehouse name.
Now query MySQL directly through DuckDB:
await show("top products, straight from MySQL", `
SELECT product_name,
sum(quantity) AS units,
round(sum(line_total), 2) AS revenue
FROM shop.order_lines
GROUP BY product_name
ORDER BY revenue DESC
LIMIT 5
`);--- top products, straight from MySQL ---
┌────────────────────────┬───────┬─────────────┐
│ product_name │ units │ revenue │
├────────────────────────┼───────┼─────────────┤
│ Durango SXT RWD │ 19 │ 702999.81 │
│ Dodge Hornet GT Plus │ 17 │ 424999.83 │
│ MotoGP CI.H1 │ 22 │ 329999.78 │
│ 300 Touring │ 9 │ 260999.91 │
│ Pacifica Touring │ 8 │ 255999.92 │
└────────────────────────┴───────┴─────────────┘No export, no ETL job, no copy step. DuckDB pushes what it can down to MySQL and executes the rest in its own engine.
Then materialise an aggregate into Iceberg:
await db.run(`CREATE SCHEMA IF NOT EXISTS lake.analytics`);
await db.run(`DROP TABLE IF EXISTS lake.analytics.product_revenue`);
await db.run(`
CREATE TABLE lake.analytics.product_revenue AS
SELECT product_id,
any_value(product_name) AS product_name,
count(DISTINCT order_id) AS orders,
sum(quantity) AS units,
round(sum(line_total), 2) AS gross_revenue,
round(avg(discount_pct), 2) AS avg_discount_pct
FROM shop.order_lines
GROUP BY product_id
`);
await show("same aggregate, now read back out of Iceberg", `
SELECT product_name, orders, units, gross_revenue
FROM lake.analytics.product_revenue
ORDER BY gross_revenue DESC
LIMIT 5
`);--- same aggregate, now read back out of Iceberg ---
┌────────────────────────┬────────┬───────┬───────────────┐
│ product_name │ orders │ units │ gross_revenue │
├────────────────────────┼────────┼───────┼───────────────┤
│ Durango SXT RWD │ 6 │ 19 │ 702999.81 │
│ Dodge Hornet GT Plus │ 4 │ 17 │ 424999.83 │
│ MotoGP CI.H1 │ 5 │ 22 │ 329999.78 │
│ 300 Touring │ 5 │ 9 │ 260999.91 │
│ Pacifica Touring │ 2 │ 8 │ 255999.92 │
└────────────────────────┴────────┴───────┴───────────────┘That is a real Apache Iceberg table. Open the MinIO console at http://localhost:9001 (minioadmin / minioadmin) and you will find Parquet data files and a metadata/ folder of JSON and Avro manifests under warehouse/analytics/product_revenue/.
Step 5: The Things Parquet Alone Cannot Do
If Iceberg were only "Parquet in a folder," it would not be worth a catalog container. Here is what the format actually buys you.
Row-level mutation. A pile of Parquet files has no UPDATE.
await db.run(`
INSERT INTO lake.analytics.product_revenue
VALUES (9999, 'Gift Card', 12, 12, 480.00, 0.00)
`);
await db.run(`
UPDATE lake.analytics.product_revenue SET avg_discount_pct = 5.00 WHERE product_id = 9999
`);Both commit atomically. A reader either sees the whole change or none of it, and never a half-written table.
A queryable history. Every commit is a snapshot:
await show("snapshot history", `
SELECT sequence_number, snapshot_id, timestamp_ms
FROM iceberg_snapshots('lake.analytics.product_revenue')
ORDER BY sequence_number
`);--- snapshot history ---
┌─────────────────┬───────────────────────┬───────────────────────────┐
│ sequence_number │ snapshot_id │ timestamp_ms │
├─────────────────┼───────────────────────┼───────────────────────────┤
│ 1 │ 2538259603105727566 │ 2026-08-09 10:20:19.646 │
│ 2 │ 1160600651192946188 │ 2026-08-09 10:20:19.718 │
│ 3 │ 2080578457788763269 │ 2026-08-09 10:20:19.785 │
└─────────────────┴───────────────────────┴───────────────────────────┘Three snapshots: the CREATE TABLE AS, the INSERT, the UPDATE. Note that iceberg_snapshots() accepts the qualified catalog table name here. Point it at a bare filesystem path instead and you will hit No version was provided and no version-hint could be found, because DuckDB refuses to guess which metadata file is current.
Time travel. Query the table as it was at any snapshot:
// snapshot_id comes from our own catalog, not from user input.
const [firstSnapshot] = (
await db.runAndReadAll(`
SELECT snapshot_id FROM iceberg_snapshots('lake.analytics.product_revenue')
ORDER BY sequence_number LIMIT 1
`)
).getRowsJson()[0];
await show("row count: first snapshot vs now", `
SELECT
(SELECT count(*) FROM lake.analytics.product_revenue AT (VERSION => ${firstSnapshot})) AS at_first_snapshot,
(SELECT count(*) FROM lake.analytics.product_revenue) AS latest
`);--- row count: first snapshot vs now ---
┌───────────────────┬────────┐
│ at_first_snapshot │ latest │
├───────────────────┼────────┤
│ 189 │ 190 │
└───────────────────┴────────┘AT (TIMESTAMP => ...) works too. This is the feature that turns "the dashboard numbers changed and nobody knows why" from an archaeology project into a query. It is also how you reproduce a model training run six months later against exactly the data it saw.
And it all still joins. The lakehouse table and the live operational database are both just schemas in the same session:
await show("joining the lakehouse table back to live MySQL", `
SELECT p.product_name, p.gross_revenue, count(o.order_id) AS live_lines
FROM lake.analytics.product_revenue p
JOIN shop.order_lines o USING (product_id)
GROUP BY p.product_name, p.gross_revenue
ORDER BY p.gross_revenue DESC
LIMIT 5
`);--- joining the lakehouse table back to live MySQL ---
┌────────────────────────┬───────────────┬────────────┐
│ product_name │ gross_revenue │ live_lines │
├────────────────────────┼───────────────┼────────────┤
│ Durango SXT RWD │ 702999.81 │ 6 │
│ Dodge Hornet GT Plus │ 424999.83 │ 4 │
│ MotoGP CI.H1 │ 329999.78 │ 5 │
│ 300 Touring │ 260999.91 │ 5 │
│ Pacifica Touring │ 255999.92 │ 2 │
└────────────────────────┴───────────────┴────────────┘Historical data in object storage joined against live transactional data, in one SQL statement, from a Node.js process, with no cluster running. That is the whole pitch.
Troubleshooting: The Errors You Will Actually Hit
| Error | Cause | Fix |
|---|---|---|
AUTHORIZATION_TYPE is 'oauth2', yet no 'secret' was provided | You gave ATTACH a local path and expected a filesystem catalog | There is no filesystem catalog. Run a REST catalog and pass ENDPOINT |
Unrecognized 'endpoint_type' (filesystem) | Only glue and s3_tables are valid ENDPOINT_TYPE values | Use a plain REST ENDPOINT, or Glue / S3 Tables |
Cannot execute statement of type "CREATE" ... attached in read-only mode | Attach string looked like a path (s3://warehouse/) | Use the bare warehouse name: ATTACH 'warehouse' |
No version was provided and no version-hint could be found | Path-based iceberg_scan / iceberg_snapshots on a directory with no version hint | Use the qualified catalog table name, or point at metadata/vN.metadata.json |
Do not know how to serialize a BigInt | DuckDB BIGINT becomes JS bigint | Use getRowObjectsJson() / getRowsJson() |
HTTP 403 reading Parquet from MinIO | S3 secret missing or wrong style | CREATE SECRET with URL_STYLE 'path' and USE_SSL false |
UPDATE/DELETE fails with a write-mode error | Table sets write.update.mode to something other than merge-on-read | DuckDB writes positional deletes only; copy-on-write is unsupported |
Taking This to Production
The code above changes surprisingly little on the way to a real deployment. What changes is what sits behind each connection string.
Object storage. Drop MinIO, point the S3 secret at real S3, GCS, or Azure Blob. On AWS, prefer an IAM role over static keys: CREATE SECRET (TYPE s3, PROVIDER credential_chain).
Catalog. The REST fixture is a test fixture and stores its state in memory. Pick a real one: AWS Glue or S3 Tables if you are on AWS (ENDPOINT_TYPE 'glue' / 's3_tables'), Lakekeeper or Apache Polaris if you want open-source and self-hosted, Unity Catalog if you are already on Databricks. The ATTACH line changes; nothing else does.
Credentials. Everything hardcoded above becomes environment variables or a secrets manager entry. Nothing goes in the repository.
Incremental loads. The tutorial does a full reload. Real pipelines pull a watermark, load the delta, and MERGE INTO the Iceberg table. Our write-up on ETL vs ELT covers where to put that transformation boundary.
Maintenance. Iceberg accumulates small files and old snapshots. Budget for periodic compaction and snapshot expiry, and set retention deliberately rather than by accident. If you operate under GDPR, note that time travel means deleted rows remain readable in old snapshots until those snapshots expire, so your retention policy and your erasure obligations have to be designed together rather than discovered together.
Orchestration. Two scripts run by hand becomes Airflow, Dagster, or a scheduled job, with retries and alerting. See how data pipelines work for the operational shape of that.
When DuckDB Is Enough, and When It Is Not
Being honest about the ceiling is more useful than pretending there is not one.
DuckDB and Iceberg are a good fit when your working set fits on one large machine, your concurrency is a handful of analysts and scheduled jobs rather than thousands of dashboard users, and you want to keep infrastructure and spend small. That describes most companies below roughly the mid-market line, and plenty above it.
You will outgrow it when you need heavy concurrent multi-user serving, distributed compute across genuinely large data, streaming ingestion at scale, or the governance and lineage tooling that comes with a managed platform. At that point the fact that you chose Iceberg pays off: the tables stay exactly where they are, and Databricks, Snowflake, Trino, or Spark reads them in place. Our comparison of Databricks vs Snowflake covers that decision when you get there.
The mistake worth avoiding is the opposite one: buying the distributed platform first, for data volumes that a laptop would handle, and then spending a year building a team around it.
Frequently Asked Questions
Can DuckDB write Iceberg tables without a catalog?
No. As of DuckDB 1.5.5, writing Iceberg requires an attached catalog, and the only supported catalog types are REST, AWS Glue, and S3 Tables. The path-based iceberg_scan() interface is read-only. If you want a catalog-free local lakehouse, DuckLake does that, but it is not Iceberg and other engines will not read it.
Which DuckDB version added Iceberg write support?
1.4.0, released September 2025, added CREATE TABLE, CREATE TABLE AS SELECT, and INSERT. 1.4.2 added UPDATE and DELETE via merge-on-read positional deletes. 1.5.3 added MERGE INTO, full ALTER TABLE schema evolution, bucket and truncate partition transforms, and Iceberg V3 support.
Do I need Spark to use Apache Iceberg?
No. Iceberg is a table format specification, not a Spark feature. DuckDB, Trino, Flink, ClickHouse, Snowflake, and Databricks can all read and, to varying degrees, write it. This tutorial uses no JVM code you write yourself; the only JVM in play is inside the catalog container.
Is DuckDB production-ready for analytics?
For single-node analytical workloads, yes, and it is widely used that way. The Iceberg extension specifically is still labelled experimental by its maintainers, so test your write patterns before depending on them. Read paths are considerably more mature than write paths.
Can I use PostgreSQL instead of MySQL?
Yes. Swap INSTALL mysql for INSTALL postgres and the attach string for ATTACH 'dbname=shop host=127.0.0.1 user=shop' AS shop (TYPE postgres, READ_ONLY). Everything downstream is identical. DuckDB also has extensions for SQLite and for scanning Parquet, CSV, and JSON directly.
How does this compare to just using Parquet files?
Parquet gives you columnar storage and compression. Iceberg adds ACID commits, row-level updates and deletes, schema evolution without rewrites, snapshot history and time travel, partition evolution, and a catalog that lets multiple engines agree on what the table currently is. If you only ever append and only ever read with one engine, Parquet may be enough. Everything past that is what the format is for. Our deep dive on Apache Parquet covers the storage layer underneath.
What does it cost to run this in production?
The compute is a container or a Lambda-sized process rather than a cluster, and storage is object-storage pricing. The dominant cost for most teams that adopt this pattern is engineering time, not infrastructure, which is the inverse of the usual warehouse bill.
Wrapping Up
Twenty minutes, three containers, two scripts. You now have a pipeline that pulls from an API into MySQL, queries that MySQL from DuckDB with no export step, writes an Apache Iceberg table, mutates it transactionally, and time travels through its history.
The important part is not that it runs locally. It is that the same code, with different connection strings, runs on S3 and Glue. The Iceberg tables you write are readable by every major engine, which means this is a starting point you can grow out of without a migration project.
The two things worth remembering when you build on this: writes need a catalog, and the attach string must be the warehouse name, not a path. Those two facts account for most of the time people lose on their first DuckDB Iceberg project.
---
Building this for real?
Lucent Innovation's data engineering services team designs and ships lakehouse platforms on Apache Iceberg, Databricks, and DuckDB, from first pipeline to production governance. We handle the parts this tutorial deliberately skipped: incremental loads, catalog selection, compaction and retention, orchestration, access control, and cost design.
If you are weighing platform options before you commit, our modern data architecture consulting engagement is built for exactly that decision. If you need engineers rather than a project, you can hire data engineers from our team, and our guide to hiring for Databricks covers the engagement models and what each actually costs.
Either way, start a conversation and we will tell you honestly whether you need a platform or just a better query.