CLI Reference¶

iceberg-ops ships one console script, iceberg-ops. Global options (catalog, namespace, paths, config file) go before the command; each command’s own options go after it:

iceberg-ops [GLOBAL OPTIONS] COMMAND [OPTIONS]
iceberg-ops --catalog sqlite --namespace sales migrate --target 0002
iceberg-ops --version

The command reference at the end of this page is generated from the installed Click commands, so it shows the same options and help text as iceberg-ops COMMAND --help.

The commands fall into three groups:

Project commands – status, current, migrate, rollback, stamp, history, plan. They need a catalog and --namespace, from flags or the config file.
Scaffolding – init and revision need no catalog. revision --autogenerate does, because it loads the existing table (see Autogenerate).
Standalone commands – migrate-data (see Cross-Table Data Migration) and run-policies (experimental, see Polaris Policy Executor (experimental)) take their own catalog options; the global catalog options and the config file’s catalog settings do not apply to them.

Every project command except history imports and executes the migration files it discovers, with your catalog credentials, so plan and status are not safe on files you have not reviewed. revision --target-schema-arrow imports the module it names. See Trust model.

Choosing a catalog¶

--catalog sqlite: A local SQLite catalog. Without --db-path / --warehouse it lives in local_data/iceberg_catalog.db and local_data/iceberg_warehouse next to the config file, or in the current directory when there is no config file.
--catalog-name NAME: A catalog defined in .pyiceberg.yaml or in PYICEBERG_CATALOG__<NAME>__* environment variables, opened with pyiceberg.catalog.load_catalog().
--catalog glue|rest|hive|s3tables: Loads the catalog named after the kind in the same way, so PYICEBERG_CATALOG__REST__URI configures --catalog rest. Combined with --catalog-name, it opens that named catalog as that kind.

The CLI does not accept the memory kind: its state is gone when the command exits. --db-path applies to sqlite only, and a named catalog cannot use the sqlite kind.

Configuration file¶

By default the CLI reads iceberg-ops.yaml from the current directory; --config PATH reads another file. A --config path that does not exist is a usage error, except for init, which creates it. iceberg-ops init writes a commented starter file; a minimal one looks like this:

catalog: sqlite
namespace: sales

Rules:

The keys are catalog, catalog_name, namespace, db_path, warehouse and migrations_dir. Every value must be a string, so quote numbers (namespace: "2024"). Unknown keys print a warning and are ignored.
Command-line flags override the file. A --catalog-name flag replaces the file’s catalog and db_path; a --catalog flag replaces the file’s catalog_name. The file’s db_path and warehouse apply only when the flags select the same catalog as the file.
Setting both catalog: sqlite and catalog_name in the file is a usage error. To use a named catalog, delete the catalog: sqlite line that init writes.
Relative db_path, migrations_dir and (for sqlite) warehouse values resolve against the directory that holds the file. Relative paths given as flags resolve against the current directory. migrations/ defaults to sitting next to the file.
The file is read as UTF-8; a byte-order mark is allowed.

Exit codes¶

Code	Meaning
`0`	Success.
`1`	A runtime failure, reported as one `Error:` line instead of a traceback: the catalog cannot be opened (missing extra, missing property, connection, permission or server error), a table or namespace is missing, a commit conflicts, a migration fails or cannot be loaded, a `downgrade()` raises `NotImplementedError`, a checkpoint is corrupt or already completed, a revision file already exists, or `migrate-data --verify partition_hash` runs on a destination with non-identity partitions (`Verify error:`). `run-policies` prints `Polaris error:` when the Polaris request fails. Set `ICEBERG_OPS_TRACEBACK=1` to see the original traceback instead of the `Error:` line.
`2`	A usage error: an unknown or malformed option, a bad config file, a missing catalog or `--namespace`, a missing migrations directory, a `--target` that names no discovered migration, `rollback --steps` combined with `--target`, a source and destination that are the same table in `migrate-data`, an invalid Polaris token in `run-policies`, or, for `revision --autogenerate`, a target table that does not exist or a target schema that cannot be read or compared. Also returned when `migrate-data --verify` reports a mismatch and when `run-policies` has at least one failed policy.

--target takes a migration version with or without leading zeros (2 means 0002); rollback --target 0 rolls back every applied migration. rollback undoes one migration by default; --steps must be at least 1.

Command reference¶

iceberg-ops¶

Alembic-style schema migrations for Apache Iceberg.

Project commands (status, current, migrate, rollback, stamp, history, plan) need a catalog and --namespace, from flags or the config file. Choose the catalog with --catalog (sqlite keeps a local catalog under local_data/) or with --catalog-name for a catalog defined in .pyiceberg.yaml or PYICEBERG_CATALOG__<NAME>__* environment variables. init and revision need no catalog (revision --autogenerate does); migrate-data and run-policies take their own catalog options.

Set ICEBERG_OPS_TRACEBACK=1 to see full tracebacks.

Usage

iceberg-ops [OPTIONS] COMMAND [ARGS]...

Options

--version¶: Show the version and exit.

--catalog <catalog>¶

Catalog kind. sqlite keeps a local catalog under local_data/.

Options:: sqlite | glue | rest | hive | s3tables

--catalog-name <catalog_name>¶: Catalog defined in .pyiceberg.yaml or PYICEBERG_CATALOG__<NAME>__* env vars.

--namespace <namespace>¶: Target namespace.

--db-path <db_path>¶: SQLite catalog DB path (sqlite only).

--warehouse <warehouse>¶: Warehouse directory (sqlite) or warehouse location/URI (remote catalogs).

--migrations-dir <migrations_dir>¶: Migrations directory (default: migrations/ next to the config file).

--config <config>¶: Config file path (default: iceberg-ops.yaml).

current¶

Alias of status - show the current migration state for the namespace.

Usage

iceberg-ops current [OPTIONS]

history¶

Print migration history for the namespace.

Usage

iceberg-ops history [OPTIONS]

init¶

Scaffold a new iceberg-ops project in the current directory.

Creates migrations/ (with __init__.py), an iceberg-ops.yaml stub, and appends .gitignore rules for local state. The group --config and --migrations-dir options choose other locations; migrations/ goes next to the config file by default. Idempotent: running twice without --force leaves existing files untouched. Does not require --catalog or --namespace.

Usage

iceberg-ops init [OPTIONS]

Options

--force¶: Overwrite an existing config file.

migrate¶

Apply pending migrations.

Usage

iceberg-ops migrate [OPTIONS]

Options

--target <target>¶: Apply up to (and including) this version. Version, e.g. 0002 (leading zeros optional).

migrate-data¶

Cross-table data migration: stream rows between Iceberg tables.

Reads from --src-catalog/--src-namespace.--src-table and writes to --dst-catalog/--dst-namespace.--dst-table. Each commit holds at most --batch-rows source rows and is one catalog commit, but can create several destination snapshots: one for append, two for overwrite_partition, and up to three for upsert (three per 1,000-key sub-batch when a composite-key batch is split; an upsert may also add one empty schema snapshot first). To undo a run, roll the destination back to the “Destination snapshot before” ID the summary prints. The destination table must already exist and must not be the source table.

The destination catalog defaults to the source: useful for cross-namespace migration in one catalog. Pass --dst-catalog or --dst-catalog-name to cross catalogs (e.g. sqlite source, REST destination via --dst-catalog-name).

Schema reconciliation is opt-in: by default the source and dest must have matching schemas. Use --column-mapping, --on-missing-in-source, and --on-missing-in-dest to enable renames, null-fill, or auto-add.

Pass --checkpoint PATH to make the migration resumable: a crashed run picks up at the last committed batch on the next invocation.

Pass --dry-run to estimate row + byte counts without writing.

Pass --verify METHOD to run a post-migration consistency check (row_count, partition_hash, or sample).

Usage

iceberg-ops migrate-data [OPTIONS]

Options

--src-catalog <src_catalog>¶

Source catalog kind. Give this or --src-catalog-name.

Options:: sqlite | glue | rest | hive | s3tables

--src-catalog-name <src_catalog_name>¶: Source catalog defined in .pyiceberg.yaml or PYICEBERG_CATALOG__<NAME>__* env vars.

--src-namespace <src_namespace>¶: Required Source namespace.

--src-table <src_table>¶: Required Source table name.

--src-db-path <src_db_path>¶: SQLite DB path for source (when src-catalog=sqlite).

--src-warehouse <src_warehouse>¶: Warehouse directory (sqlite) or warehouse location (remote) for source.

--dst-catalog <dst_catalog>¶

Destination catalog kind. Defaults to the source catalog.

Options:: sqlite | glue | rest | hive | s3tables

--dst-catalog-name <dst_catalog_name>¶: Destination catalog from .pyiceberg.yaml or PYICEBERG_CATALOG__<NAME>__* env vars. Defaults to --src-catalog-name when --dst-catalog is not given.

--dst-namespace <dst_namespace>¶: Required Destination namespace.

--dst-table <dst_table>¶: Destination table name. Defaults to --src-table.

--dst-db-path <dst_db_path>¶: SQLite DB path for destination. Defaults to --src-db-path for the same catalog.

--dst-warehouse <dst_warehouse>¶: Warehouse for destination. Defaults to --src-warehouse for the same catalog.

--row-filter <row_filter>¶: PyIceberg row filter expression applied to the source scan.

--source-snapshot <source_snapshot>¶: Pin the source read to this snapshot ID (default: current).

--batch-rows <batch_rows>¶

Target source rows per write commit. Bounds the rows in each commit, not peak memory.

Default:: 50000

--write-mode <write_mode>¶

How each chunk is written to the destination. overwrite_partition needs an identity-partitioned destination and a migration that fits in one commit (at most --batch-rows source rows).

Default:: 'append'
Options:: append | upsert | overwrite_partition

--upsert-keys <upsert_keys>¶: Comma-separated columns. Required when --write-mode=upsert.

--checkpoint <checkpoint>¶: Checkpoint JSON path for resumability.

--column-mapping <SRC:DST>¶: Source-to-dest column rename. Repeatable.

--on-missing-in-source <on_missing_in_source>¶

Policy when a destination column has no source counterpart.

Default:: 'error'
Options:: error | null | skip

--on-missing-in-dest <on_missing_in_dest>¶

Policy when a source column has no destination counterpart.

Default:: 'error'
Options:: error | auto_add

--dry-run¶: Pre-flight only: count rows + bytes, do not write.

--verify <verify>¶

Run a post-migration check and print the result. partition_hash needs an identity-partitioned or unpartitioned destination and reads both tables in full; sample compares the first --sample-size rows of each side and can report a false mismatch on larger tables. Meant for an append into an empty destination.

Options:: row_count | partition_hash | sample

--sample-size <sample_size>¶

Sample size for --verify=sample.

Default:: 100

--quiet¶: Suppress per-batch progress output.

plan¶

Preview pending migrations without applying.

Usage

iceberg-ops plan [OPTIONS]

Options

--target <target>¶: Show only migrations up to this version. Version, e.g. 0002 (leading zeros optional).

revision¶

Create a new migration file with the next version number.

Without --autogenerate: picks max(NNNN) + 1 from existing files in --migrations-dir (or the <namespace>/ subdir if it exists) and writes a BaseMigration skeleton. Does not require --catalog; --namespace is consulted only to pick a subdir.

With --autogenerate: loads --target-table from the catalog, diffs it against the target schema (YAML SchemaContract or Python pa.Schema), and writes a migration that closes the gap. Requires a catalog (--catalog, --catalog-name or the config file) and --namespace to load the existing table. A bare --target-table stays bare in the file, so the migration applies to whichever --namespace it is run against. Steps that cannot run or cannot be reversed make upgrade() / downgrade() raise before any step runs; the command lists them. Always review the generated file before running it.

Usage

iceberg-ops revision [OPTIONS]

Options

-m, --message <message>¶: Required Short description of the migration; used as the filename slug.

--autogenerate, --no-autogenerate¶: Diff an existing Iceberg table against a target schema and emit matching add_column / rename_column / type_change / delete_column ops. Requires --target-table and exactly one of --target-schema-yaml / --target-schema-arrow.

--target-table <target_table>¶: Existing table to compare against (required with --autogenerate). Use a bare name (e.g. ‘orders’): it is loaded from --namespace now and resolved against the --namespace given at migrate time. A dotted name pins the generated migration to that one table.

--target-schema-yaml <target_schema_yaml>¶: Path to a YAML SchemaContract describing the desired schema.

--target-schema-arrow <target_schema_arrow>¶: Importable target schema in ‘module:attr’ form. The attr must resolve to a pa.Schema (e.g. ‘myproj.schemas:USERS_TARGET’); the module must be importable from the current directory or the environment.

--rename-hint <rename_hint>¶: old:new pair telling autogen to emit a rename_column op instead of drop+add. Repeatable (e.g. --rename-hint name:full_name --rename-hint score:rating).

rollback¶

Roll back applied migrations.

Usage

iceberg-ops rollback [OPTIONS]

Options

--steps <steps>¶: Number of migrations to undo (default 1). Not combinable with --target.

--target <target>¶: Roll back until this version is the latest applied. Version, e.g. 0002 (leading zeros optional).

run-policies¶

[experimental] Fetch Polaris policies for a table and dispatch each one.

Experimental: the policy requests do not yet match the released Polaris policy API, so released Polaris servers are not supported. See https://namitkewat.github.io/iceberg-ops/polaris.html.

Fetches the table’s policies from Polaris for --target-namespace.--target-table, then loads the same table from --catalog / --catalog-name and dispatches each policy. Only system.snapshot-expiry is executed (it must have “enable”: true); every other policy type is reported as skipped.

Auth is either --polaris-token (pre-fetched bearer) or the OAuth2 client-credentials pair (--polaris-client-id plus --polaris-client-secret). Exactly one of the two must be set. Prefer the ICEBERG_OPS_POLARIS_* environment variables over flags for secrets.

Use --dry-run to forward dry_run=True to each maintenance op so the table is not mutated.

Usage

iceberg-ops run-policies [OPTIONS]

Options

--polaris-url <polaris_url>¶: Required Polaris base URL, e.g. https://polaris.example.com.

--polaris-catalog <polaris_catalog>¶: Required Polaris catalog name.

--polaris-token <polaris_token>¶: Pre-fetched bearer token. Mutually exclusive with --polaris-client-id. Prefer the env var: argv is visible in process listings and CI logs.

--polaris-client-id <polaris_client_id>¶: OAuth2 client ID. Pair with --polaris-client-secret.

--polaris-client-secret <polaris_client_secret>¶: OAuth2 client secret. Pair with --polaris-client-id. Prefer the env var: argv is visible in process listings and CI logs.

--polaris-scope <polaris_scope>¶: OAuth2 scope (default: PRINCIPAL_ROLE:ALL).

--target-namespace <target_namespace>¶: Required Target namespace (passed to Polaris and used for the local table load).

--target-table <target_table>¶: Required Target table name (without namespace prefix).

--catalog <local_catalog>¶

Catalog kind for the table the policies act on. Give this or --catalog-name.

Options:: sqlite | glue | rest | hive | s3tables

--catalog-name <local_catalog_name>¶: Catalog for the table, defined in .pyiceberg.yaml or PYICEBERG_CATALOG__<NAME>__*.

--db-path <db_path>¶: SQLite DB path for the catalog (sqlite only).

--warehouse <warehouse>¶: Warehouse directory (sqlite) or warehouse location/URI (remote catalogs).

--dry-run¶: Forward dry_run=True to each maintenance op (no mutations).

Environment variables

ICEBERG_OPS_POLARIS_TOKEN: Provide a default for --polaris-token

ICEBERG_OPS_POLARIS_CLIENT_ID: Provide a default for --polaris-client-id

ICEBERG_OPS_POLARIS_CLIENT_SECRET: Provide a default for --polaris-client-secret

stamp¶

Mark migrations as applied without running them.

Used to adopt iceberg-ops on an existing Iceberg deployment whose schema was created out-of-band. Every discovered migration that is not currently applied (never run, failed or rolled back) gets an applied row. A failed version is therefore marked applied without being re-run, even if its upgrade() stopped halfway: check such versions with status before stamping. No upgrade() is invoked and no user tables are created.

Usage

iceberg-ops stamp [OPTIONS]

Options

--target <target>¶: Mark migrations applied up to (and including) this version. Version, e.g. 0002 (leading zeros optional).

status¶

Show migration status for the namespace.

Usage

iceberg-ops status [OPTIONS]