Quick Start

This page builds a small project against a local SQLite catalog: one hand-written migration, one autogenerated migration, then migrate, status and rollback. Every command runs as shown.

Install

python -m venv .venv
source .venv/bin/activate            # Windows PowerShell: .venv\Scripts\Activate.ps1
pip install "iceberg-ops[sqlite]"

The [sqlite] extra brings the local SQLite and in-memory catalogs. Other extras:

pip install "iceberg-ops[glue]"         # AWS Glue
pip install "iceberg-ops[hive]"         # Hive Metastore
pip install "iceberg-ops[s3tables]"     # AWS S3 Tables (REST with SigV4 signing)
pip install "iceberg-ops[transforms]"   # writing to bucket / truncate / time-partitioned tables
pip install "iceberg-ops[polaris]"      # experimental Apache Polaris policy executor

REST catalogs (Apache Polaris, Nessie, Lakekeeper and others) need no extra. Compatibility and limits lists the supported Python, PyIceberg and PyArrow versions. To work on iceberg-ops itself, see CONTRIBUTING.md.

Scaffold a project

mkdir demo && cd demo
iceberg-ops init

This writes:

  • iceberg-ops.yaml – a config file stub that selects the local SQLite catalog (catalog: sqlite) and the default namespace

  • migrations/__init__.py – so the directory is a real Python package

  • a marked block in .gitignore that ignores local_data/, where the SQLite catalog lives by default

Write your first migration

iceberg-ops revision -m "create orders"

This creates migrations/0001_create_orders.py. Replace its contents with:

# migrations/0001_create_orders.py
import pyarrow as pa

from iceberg_ops.migration import BaseMigration, MigrationContext


class Migration(BaseMigration):
    version = "0001"
    description = "Create orders table"

    def upgrade(self, ctx: MigrationContext) -> None:
        ctx.create_table_if_not_exists(
            "orders",
            pa.schema(
                [
                    pa.field("order_id", pa.int64(), nullable=False),
                    pa.field("order_total", pa.float64()),
                    pa.field("created_at", pa.timestamp("us", tz="UTC")),
                ]
            ),
        )

    def downgrade(self, ctx: MigrationContext) -> None:
        ctx.drop_table_if_exists("orders")

Migration files are namespace-agnostic: pass bare table names to the MigrationContext helpers and the runner qualifies them with the active namespace at apply time, so the same file works for sales, staging or finance. version must match the file’s NNNN prefix.

Apply it to a local SQLite catalog (demo.db, data files under warehouse/) in the sales namespace. Global options go before the subcommand:

iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales migrate

Autogenerate the next migration

Describe the schema you want in contracts/orders.yaml:

# contracts/orders.yaml
columns:
  - {name: order_id, type: long, required: true}
  - {name: order_amount, type: double}  # renamed from order_total
  - {name: created_at, type: timestamptz}
  - {name: status, type: string}  # new column

Let iceberg-ops diff the live table against it and write the migration:

iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales \
  revision -m "rename total, add status" --autogenerate \
  --target-table orders --target-schema-yaml contracts/orders.yaml \
  --rename-hint order_total:order_amount

The command writes migrations/0002_rename_total_add_status.py. Review it before running it. Its migration class reads:

class Migration(BaseMigration):
    version = '0002'
    description = 'rename total, add status'

    def upgrade(self, ctx: MigrationContext) -> None:
        target_table = 'orders'

        ctx.enforce_rename(target_table, 'order_total', 'order_amount')
        ctx.enforce_column(target_table, 'status', ice_types.StringType())

    def downgrade(self, ctx: MigrationContext) -> None:
        target_table = 'orders'

        ctx.delete_column(target_table, 'status')
        ctx.enforce_rename(target_table, 'order_amount', 'order_total')

Without the --rename-hint, the rename would appear as a drop plus an add. A step that cannot be undone, such as a column drop, would make the generated downgrade() raise NotImplementedError before any step runs. See Autogenerate.

Run migrations – CLI

iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales plan
iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales migrate
iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales status
iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales rollback
iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales history

rollback undoes one migration; --steps N or --target 0001 goes further back. history prints the latest recorded status of each version. status, plan and history only print, and exit 0 even when a migration is recorded as failed.

To drop the flags, put them in iceberg-ops.yaml. Relative paths in the file resolve against the directory that holds it:

# iceberg-ops.yaml
catalog: sqlite
namespace: sales
db_path: demo.db
warehouse: warehouse

Then iceberg-ops migrate, iceberg-ops status and the other commands need no options. Without db_path and warehouse, the SQLite catalog lives under local_data/ next to the config file.

Run migrations – Python API

from iceberg_ops.core.catalog import CatalogFactory
from iceberg_ops.runner import MigrationRunner

catalog = CatalogFactory.build("sqlite", db_path="demo.db", warehouse_dir="warehouse")
runner = MigrationRunner(catalog, namespace="sales", migrations_dir="migrations")

runner.status()  # show pending vs applied
runner.migrate()  # apply all pending migrations
runner.rollback(steps=1)  # undo the last migration
runner.history()  # latest recorded status of each version

The runner methods print their report and return None.

Use your own catalog

--catalog-name (or catalog_name: in iceberg-ops.yaml) opens a catalog defined in .pyiceberg.yaml or PYICEBERG_CATALOG__<NAME>__* environment variables:

iceberg-ops --catalog-name prod --namespace sales plan
iceberg-ops --catalog-name prod --namespace sales migrate

From Python, pass any PyIceberg catalog, for example pyiceberg.catalog.load_catalog("prod"), to MigrationRunner. Only the SQLite and in-memory catalogs are integration-tested; see Compatibility and limits.

Adopt on an existing deployment

If your tables were created out-of-band, stamp records every discovered migration as applied without invoking upgrade():

iceberg-ops --catalog-name prod --namespace sales stamp
iceberg-ops --catalog-name prod --namespace sales stamp --target 0002   # or stop at a version

Later migrations apply normally on top.

Where to next