iceberg-ops

Alembic-style schema migrations for Apache Iceberg. Versioned, reviewable table changes: numbered upgrade() / downgrade() revisions, autogenerate, stamp, and a migration history stored as an Iceberg table in your own catalog. Pure Python on PyIceberg: no Spark, no JVM.

iceberg-ops brings the migration workflow that application teams use for relational databases to Apache Iceberg tables. Each schema change is a numbered Python file that you review in a pull request, apply with migrate and undo with rollback. The status of each migration is recorded in a history table that lives next to your data.

The CLI is iceberg-ops; the Python package is iceberg_ops.

60-second demo

pip install "iceberg-ops[sqlite]"
mkdir demo && cd demo
iceberg-ops init
iceberg-ops revision -m "create orders"   # then fill in upgrade() / downgrade()
iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales migrate

# Describe the next schema in contracts/orders.yaml, then let iceberg-ops write the migration:
iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales \
  revision -m "rename total, add status" --autogenerate \
  --target-table orders --target-schema-yaml contracts/orders.yaml \
  --rename-hint order_total:order_amount

iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales plan
iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales migrate
iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales status
iceberg-ops --catalog sqlite --db-path demo.db --warehouse warehouse --namespace sales rollback

Quick Start walks through the same steps with the contents of the migration file and the schema contract.

What iceberg-ops is, and isn’t

It is a migration framework (files, history, rollback, stamp, autogenerate) for Iceberg schemas, partition specs, sort orders and table properties, shipped as a Python library and CLI on top of PyIceberg. The Python API takes any PyIceberg catalog; the CLI opens a local SQLite catalog or any catalog defined in .pyiceberg.yaml or PYICEBERG_CATALOG__* environment variables.

It isn’t:

• a query engine or a managed maintenance service;

• a merge-on-read writer: the rewrites it performs are copy-on-write, and it does not write delete files or deletion vectors. Compaction does not support tables that already contain them and does not check for them, so do not run it on such tables;

• a lock service: concurrent runs for one namespace are not guarded. A second concurrent run is detected when it records history (ConcurrentMigrationError), but its upgrade() may already have run. Run migrations for a namespace from a single job;

• a partition-rewrite engine: migrate-data --write-mode overwrite_partition is refused unless the whole migration fits in one commit.

Also included, secondary to the migration workflow:

• Cross-table data migration (experimental): iceberg-ops migrate-data streams rows between Iceberg tables, in one catalog or across catalogs. See Cross-Table Data Migration.

• Maintenance helpers (basic): snapshot expiry, a health report, and compaction and orphan removal for tables on the local filesystem. See Storage Maintenance.

• Apache Polaris policy executor (experimental): runs system.snapshot-expiry policies; it does not yet work against a released Polaris server. See Polaris Policy Executor (experimental).

Compatibility and limits lists the supported versions, catalogs and limits.

User Guide

• Quick Start
  • Install
  • Scaffold a project
  • Write your first migration
  • Autogenerate the next migration
  • Run migrations – CLI
  • Run migrations – Python API
  • Use your own catalog
  • Adopt on an existing deployment
  • Where to next
• Schema Migrations
  • Migration file conventions
  • MigrationContext API
  • Lifecycle hooks
  • Example: pre_upgrade quality gate
  • Downgrades
  • Legacy archival on incompatible type change
  • Migration history
  • Concurrent runs
  • Trust model
  • Running migrations from Python
  • Schema contracts
• Autogenerate
  • Diff functions
  • Renderer
  • CLI usage
  • Python API
  • Limitations
• Architecture
  • Where iceberg-ops fits
  • Capability surface
  • Catalog backends
  • Self-managed metadata namespace
  • Namespace-scoped runner
  • Local-filesystem-only operations
• Compatibility and limits
  • Python and dependencies
  • Extras
  • Catalogs
  • Filesystems
  • Format versions, delete files and deletion vectors
  • Concurrency
  • Experimental and frozen modules
  • Upstream PyIceberg issues
  • Windows
• Cross-Table Data Migration
  • Three composable pieces
  • Write modes
  • Checkpoints and resume
  • Upserts
  • Partitioned destinations
  • Verify methods
  • Undoing a migration
  • CLI usage
  • Python API
  • Testing seam
• Storage Maintenance
  • The four helpers
  • Compaction example
  • Snapshot expiry example
  • Orphan cleanup example
  • Health report example
  • Known limitations
  • Filesystem support matrix
• Polaris Policy Executor (experimental)
  • Pieces
  • Authentication
  • Policy types
  • Custom policy types
  • CLI usage
  • Python API
  • Testing seam
• CLI Reference
  • Choosing a catalog
  • Configuration file
  • Exit codes
  • Command reference

API Reference

• API Reference
  • Schema migrations
  • Core helpers
  • Experimental
  • Testing

Project

• Changelog

Project links

• Source code and issues

• Contributing

• Roadmap

• Security policy

• Code of conduct

For coding agents:

• AGENTS.md is the guide for agents that change the repository.

• agent-skills/iceberg-ops-migrations/SKILL.md is a skill that teaches a coding agent to write, review and run iceberg-ops migrations.

• llms.txt and llms-full.txt are a plain-text index of this site and the full text of every page.

Indices

• Index

• Module Index

• Search Page

License and trademarks

iceberg-ops is released under the MIT License.

Apache Iceberg, Iceberg, Apache Polaris and Apache are trademarks of the Apache Software Foundation. iceberg-ops is not affiliated with or endorsed by the ASF.