Autogenerate

iceberg-ops revision --autogenerate compares an existing Iceberg table against a target schema and writes a migration file whose upgrade() closes the gap and whose downgrade() undoes it where the prior state is known.

• From the CLI, the target is a YAML SchemaContract or an importable pa.Schema (module:attr), and only columns are diffed.

• From Python, the target can also be a pyiceberg.schema.Schema, and spec_diff_to_ops / sort_order_diff_to_ops / properties_diff_to_ops diff the partition spec, sort order and table properties.

Generated files always carry a “Review every line before running” warning: autogenerate is a starting point, not a finished implementation. Rename detection in particular requires explicit hints, and steps that cannot run or cannot be reversed make the generated method raise instead of running (see Limitations).

Generated files keep the table name exactly as you pass it. Pass a bare name (orders): the migration context qualifies it with the namespace given at migrate time, so one file serves every namespace. A dotted name (sales.orders) pins the file to that one table, and the CLI prints a warning.

Diff functions

The autogenerate machinery decomposes into four diff helpers, all exposed on the public API:

• schema_diff_to_ops(*, current, target, rename_hints=None) emits add_column / rename_column / delete_column / type_change ops. Without rename_hints, a rename is indistinguishable from a drop + add and the diff falls back to drop + add. With a pa.Schema or pyiceberg.schema.Schema target, table columns missing from the target are dropped. A non-strict SchemaContract target (the default) only reconciles the columns it lists and keeps the rest; a strict one drops unlisted columns.

• spec_diff_to_ops(*, current, target, schema) emits add_partition_field / remove_partition_field ops for partition spec evolution, matching fields by partition name. The generated call passes the transform as transforms.<Transform>(...) and the file imports from pyiceberg import transforms.

• sort_order_diff_to_ops(*, current, target, schema) emits a single set_sort_order op when the orders differ (replace semantics, not delta).

• properties_diff_to_ops(*, current, target) emits set_property / remove_property ops, preserving the previous value in args so the generated downgrade can restore it.

Renderer

render_migration_body(ops, *, target_table) returns (upgrade_body, downgrade_body); render_migration_file(*, version, description, ops, target_table) wraps them in the file template; write_autogenerated_revision(*, version, description, ops, target_table, out_path, overwrite=False) writes the file as UTF-8 with LF line endings. It raises FileExistsError when out_path exists unless overwrite=True, and it does not create missing directories. Every name, value and description is written as a Python literal, so a generated file always compiles.

Generated migrations call ctx.<method>(target_table, ...) directly – no helper = ctx.get_helper(target_table) boilerplate – because MigrationContext exposes the column, partition, sort-order and property helpers as direct methods. Generated files read like hand-written ones.

What each op renders to:

Op	upgrade()	downgrade()
add_column	ctx.enforce_column(...) with the new type (and the target doc, if any)	ctx.delete_column(...)
rename_column	ctx.enforce_rename(old, new)	ctx.enforce_rename(new, old)
delete_column	ctx.delete_column(...)	raises (re-adding gives a new, empty column)
type_change	ctx.enforce_column(...) with the wider type, for a promotion only; otherwise raises	raises (Iceberg only widens types)
add_partition_field	ctx.update_partition_spec(add_fields=[...])	ctx.update_partition_spec(remove_fields=[...])
remove_partition_field	ctx.update_partition_spec(remove_fields=[...])	raises (the old source column and transform are not recorded)
set_sort_order	ctx.replace_sort_order(...)	raises (the previous order is not recorded)
set_property	ctx.set_table_properties(...)	restores the old value, or unsets a new key
remove_property	ctx.unset_table_properties(...)	restores the old value; raises when it was not recorded

“Raises” means the whole method raises NotImplementedError right after target_table is assigned, before any ctx call, with every generated step kept below as a comment, in order. So rolling back a generated migration that drops a column fails before any of its reversible steps run, and the version stays applied.

CLI usage

Given an existing sales.orders table with columns order_id (long, required) and amount (double), and this target contract:

# contracts/orders.yaml
strict: true
columns:
  - name: order_id
    type: long
    required: true
  - name: order_amount
    type: double

the command below writes the next migrations/NNNN_*.py file with one ctx.enforce_rename(target_table, 'amount', 'order_amount') step and its reverse:

iceberg-ops --namespace sales revision \
  -m "align orders with v2 schema" \
  --autogenerate \
  --target-table orders \
  --target-schema-yaml contracts/orders.yaml \
  --rename-hint amount:order_amount

revision --autogenerate needs a catalog (--catalog, --catalog-name or the config file) and --namespace to load the existing table. These are group options, so they go before revision. The migrations directory must already exist (iceberg-ops init creates it); a migrations/<namespace>/ subdirectory is used when present.

Flags:

• --autogenerate / --no-autogenerate – enable diff mode.

• --target-table – existing table to compare against. Use a bare name; it is loaded from --namespace now and written into the file as typed.

• --target-schema-yaml PATH – load a SchemaContract from YAML. Contracts are read as UTF-8 and validated (strict / required must be booleans); types include decimal(P,S), fixed[N] and timestamp_ns (see Schema Migrations).

• --target-schema-arrow module:attr – import a pa.Schema (the attr must resolve to a pa.Schema). The module must be importable from the current directory or the environment.

• --rename-hint old:new – repeatable; tells the differ to emit a rename rather than drop + add. Hints are validated: an unknown column, an old name that is still in the target, a new name that already exists (swaps and chains), or two hints with the same name are usage errors (exit 2).

Plain iceberg-ops revision errors when any autogen flag is set without --autogenerate so misuse is caught early. After writing the file, the command lists the steps that make upgrade() or downgrade() raise, the table columns a non-strict contract kept, and nullability or doc differences it did not change. An invalid contract or a bad hint exits 2 without writing a file.

Python API

import tempfile
from pathlib import Path

import pyarrow as pa

from iceberg_ops.autogen import schema_diff_to_ops, write_autogenerated_revision
from iceberg_ops.core.catalog import CatalogFactory
from iceberg_ops.runner import MigrationRunner

# An existing table to compare against.
catalog = CatalogFactory.build("memory")
catalog.create_namespace("sales")
existing_table = catalog.create_table(
    "sales.orders",
    pa.schema([pa.field("order_id", pa.int64(), nullable=False), ("amount", pa.float64())]),
)

target = pa.schema(
    [
        pa.field("order_id", pa.int64(), nullable=False),
        pa.field("order_amount", pa.float64()),  # was 'amount'
    ]
)

ops = schema_diff_to_ops(
    current=existing_table.schema(),
    target=target,
    rename_hints={"amount": "order_amount"},
)

migrations_dir = Path(tempfile.mkdtemp()) / "migrations"
migrations_dir.mkdir()
write_autogenerated_revision(
    version="0001",
    description="rename amount -> order_amount",
    ops=ops,
    target_table="orders",  # bare: follows the namespace at migrate time
    out_path=migrations_dir / "0001_rename_amount.py",
)

runner = MigrationRunner(catalog, namespace="sales", migrations_dir=str(migrations_dir))
runner.migrate()  # renames amount -> order_amount
runner.rollback()  # renames it back

Limitations

• Renames need an explicit --rename-hint (rename_hints= in Python) – the differ has no other way to disambiguate a rename from a drop + add.

• Steps that cannot run as generated (a type change that is not a promotion, nested types) or cannot be reversed (column drop, type widening, partition-field removal, sort-order replacement, property removal without a recorded old value) make the affected method raise NotImplementedError before any step runs. The generated calls stay in the file as comments for manual completion.

• Type changes are one-way. Only int -> long, float -> double and decimal precision widening at the same scale are emitted; the generated downgrade of a type change always raises.

• Nested types (Struct / List / Map) are not rendered: a nested add or type change blocks the upgrade() until you write that step by hand. With a pa.Schema target, nested columns that already exist are not compared (a warning is logged, and the CLI prints one naming them) and a new nested column is rejected with ValueError.

• A generated upgrade() is idempotent and adopts columns that already exist, but downgrade() still removes every column the migration lists, even in a namespace whose table already had it. Review downgrade() before rolling back a namespace whose table differed from the one that was diffed.

• Dropping a column that the partition spec, sort order or identifier fields reference makes upgrade() raise before any step runs. The CLI passes the table’s spec, sort order and format version to the diff; from Python, pass spec=, sort_order= and format_version= to schema_diff_to_ops (identifier fields are always checked). A changed column whose name contains . raises ValueError, and so does, when format_version is given, a type the table’s format version cannot hold.

• A non-strict contract (the default) keeps table columns it does not list; set strict: true to drop them.

• Nullability and doc changes on existing columns are reported, not emitted. New columns are added as optional and keep the target’s doc.

• Each step commits on its own: if a step fails, the earlier steps stay applied and the version is recorded as failed.

• spec_diff_to_ops rejects a partition field that keeps its name but changes its source column or transform, and sort_order_diff_to_ops rejects sort fields with a non-identity transform or a non-default null order (ValueError); write those steps by hand.

• The CLI diffs columns only; use the Python diff functions for partition specs, sort orders and properties.

See Autogenerate for the full API.