Autogenerate

iceberg_ops.autogen

Autogenerate migration files from a schema diff.

This module powers iceberg-ops revision --autogenerate: it compares an existing Iceberg table against a target schema (pa.Schema, SchemaContract, or pyiceberg.schema.Schema) and produces a list of MigrationOp, which the renderer then turns into runnable migration source code.

Public API

MigrationOp

Dataclass describing one op (kind + args).

schema_diff_to_ops(*, current, target, rename_hints=None, format_version=None, spec=None, sort_order=None)()

Compute the ops that turn current into target for column schema (add / rename / delete / type_change).

spec_diff_to_ops(*, current, target, schema)()

Compute the ops that turn current partition spec into target.

sort_order_diff_to_ops(*, current, target, schema)()

Emit one set_sort_order op when the orders differ.

properties_diff_to_ops(*, current, target)()

Emit set_property / remove_property ops.

render_migration_body(ops, *, target_table)()

Translate a list[MigrationOp] into (upgrade_body, downgrade_body) Python source strings ready to drop into a BaseMigration skeleton.

render_migration_file(*, version, description, ops, target_table)()

Top-level helper that produces the full migration file source.

write_autogenerated_revision(*, version, description, ops, target_table, out_path, overwrite=False)()

Render and write the file to disk (UTF-8, LF line endings). Refuses to replace an existing file unless overwrite=True.

Scope

• Every name, value and description rendered into a generated file is written as a Python literal (repr), so a generated file always compiles and cannot run code smuggled in through a column name or a -m message.

• Schema column ops handle primitive Iceberg types. Type changes are emitted only for Iceberg type promotions (int -> long, float -> double, decimal precision widening at the same scale).

• Column names containing a dot are refused: the MigrationContext helpers read a dot as a nested struct path.

• Steps that cannot run as generated (incompatible type changes, nested types, dropping a column the partition spec, sort order or identifier fields reference) or cannot be reversed (column drops, type widening, partition-field removal, sort-order replacement, property removal with an unknown old value) make the affected upgrade / downgrade raise NotImplementedError before any step runs. The generated calls are kept as comments so the author can finish the method by hand.

• Partition specs and sort orders are diffed for programmatic use; the CLI front-end (revision --autogenerate) is schema-only.

• Nullability and doc changes on existing columns are not emitted yet.

class iceberg_ops.autogen.MigrationOp(kind, args=dict())

Bases: object

A single migration operation that the autogen renderer will emit.

Variables:

• kind (str) – One of the supported op kinds. See MigrationOpKind.

• args (dict) – Op-specific arguments. The schema is documented per-kind in the diff functions below; the renderer reads these args to generate the matching Python source.

Parameters:

• kind (Literal['add_column', 'rename_column', 'delete_column', 'type_change', 'add_partition_field', 'remove_partition_field', 'set_sort_order', 'set_property', 'remove_property'])

• args (dict[str, Any], default: dict())

kind: Literal['add_column', 'rename_column', 'delete_column', 'type_change', 'add_partition_field', 'remove_partition_field', 'set_sort_order', 'set_property', 'remove_property']

args: dict[str, Any]

iceberg_ops.autogen.properties_diff_to_ops(*, current, target)

Emit set_property and remove_property ops to reconcile the table-property dicts.

Keys present in target but absent from current (or with a different value) become set_property ops. Keys present in current but absent from target become remove_property ops; the old value is preserved in the args so the generated downgrade body can restore it.

Parameters:

• current (dict[str, str]) – table.properties of the existing table.

• target (dict[str, str]) – Desired properties.

Returns:

remove_property ops (args {"key", "old_value"}) first, then set_property ops (args {"key", "value", "old_value"}), each group sorted by key.

Return type:

list[MigrationOp]

iceberg_ops.autogen.render_migration_body(ops, *, target_table)

Translate a list of MigrationOp into upgrade and downgrade method body source strings.

Parameters:

• ops (list[MigrationOp]) – Output of schema_diff_to_ops() and friends.

• target_table (str) – Table identifier used in the generated calls. Prefer a bare name ("orders"): the migration context qualifies it with the namespace the migration runs against, so one file serves every namespace. A dotted name ("sales.orders") pins the namespace - the migration always modifies that table.

Returns:

(upgrade_body, downgrade_body) – Each is a \n-joined block of indented Python source ready to drop into a BaseMigration method.

Return type:

tuple[str, str]

Raises:

ValueError – If an op has an unknown kind.

Notes

The downgrade order is the reverse of upgrade - matches the classic schema-migration convention. When any step of a method cannot run as generated (an incompatible or nested type change, or dropping a column the table layout references, on upgrade; a column drop, type widening, partition-field removal, sort-order replacement or property removal without a recorded old value on downgrade), that method raises NotImplementedError before any step runs and keeps every generated call as a comment.

iceberg_ops.autogen.render_migration_file(*, version, description, ops, target_table)

Render a complete migration source file from autogen ops.

Combines render_migration_body() with the autogen-specific docstring (which carries the “Review every line before running” warning) and the imports the ops need.

Parameters:

• version (str) – Four-digit version, e.g. "0007".

• description (str) – Free-text description (e.g. the -m message). Stored verbatim as the description class attribute; a whitespace-collapsed copy goes into the module docstring.

• ops (list[MigrationOp]) – Ops to render.

• target_table (str) – Table identifier for the generated calls; see render_migration_body() (prefer a bare name).

Returns:

Full Python source ready to write to disk.

Return type:

str

Raises:

ValueError – If version is not four digits or an op has an unknown kind.

iceberg_ops.autogen.schema_diff_to_ops(*, current, target, rename_hints=None, format_version=None, spec=None, sort_order=None)

Compute the list of MigrationOp that turns current into target for column schema only (add / rename / delete / type_change).

Uses diff_schemas() under the hood. Renames are disambiguated by rename_hints (a mapping {old_name: new_name}); when no hint covers a pair, a column rename is emitted as drop + add.

Parameters:

• current (Schema) – Schema of the existing table - must carry stable field-IDs from a real Iceberg table (e.g. table.schema()).

• target (Schema | Schema | SchemaContract) – Desired schema. IcebergSchema is passed through; pa.Schema and SchemaContract are normalised by reusing current field-IDs for fields with the same name (or via rename hints). Columns of current missing from the target are dropped, except for a non-strict SchemaContract (strict=False, the default), which only reconciles the columns it lists and keeps the rest. Nested columns of a pa.Schema target are not compared when they already exist, and cannot be added.

• rename_hints (dict[str, str] | None, default: None) – Map of old_column_name -> new_column_name. Each entry causes a single rename_column op instead of drop + add. Not accepted with an IcebergSchema target, whose field IDs already identify renames.

• format_version (int | None, default: None) – The table’s format version (table.format_version). When given, an added or changed column whose type needs a newer version (timestamp_ns, timestamptz_ns, unknown, geometry and geography need version 3) raises ValueError. None (default) skips the check.

• spec (PartitionSpec | None, default: None) – The partition spec in effect when the deletes run (normally table.spec()). Deleting a column one of its non-void fields is sourced from is rendered as an upgrade that raises.

• sort_order (SortOrder | None, default: None) – The sort order in effect when the deletes run (normally table.sort_order()); deleting a column it sorts by is rendered as an upgrade that raises. The identifier fields of current are always checked the same way.

Returns:

Ordered list of ops. Order: removed -> renamed -> type-changed -> added; this keeps downstream renders deterministic.

Return type:

list[MigrationOp]

Raises:

• ValueError – If a rename hint is inconsistent with the two schemas (unknown old or new name, old name kept, new name already present, two hints sharing a new name), if hints are combined with an IcebergSchema target, if the target repeats a column name, if a target column type cannot be mapped, if an op names a column containing "." (the MigrationContext helpers read a dot as a nested struct path), or if format_version is given and an added or changed type needs a newer one.

• TypeError – If target is not one of the supported types.

Notes

Op kinds emitted:

• add_column - args {"name", "iceberg_type", "doc"}

• rename_column - args {"old_name", "new_name"}

• delete_column - args {"name"}, plus "referenced_by" (list of str) when the identifier fields, spec or sort_order reference the column; the renderer then blocks the upgrade.

• type_change - args {"name", "old_iceberg_type", "new_iceberg_type"}

The iceberg_type values are pyiceberg.types.IcebergType instances; the renderer converts them to source code. A type_change that is not an Iceberg type promotion is still emitted so the diff stays visible; the renderer turns it into an upgrade that raises.

Nullability and doc changes on existing columns are not emitted yet (PyIceberg supports them via UpdateSchema.make_column_optional and update_column(doc=...)). New columns are added as optional, with the target doc when one is set.

iceberg_ops.autogen.sort_order_diff_to_ops(*, current, target, schema)

Emit a single set_sort_order op when the orders differ.

The renderer translates this to ctx.replace_sort_order(target_table, sort_fields=[(col, dir), ...]) - replacing the entire sort order rather than adding / removing fields, which matches PyIceberg’s replace_sort_order semantics.

Two orders are considered identical iff they have the same number of fields, in the same order, with the same source_id, transform, direction and null_order.

Parameters:

• current (SortOrder) – table.sort_order() from the existing table.

• target (SortOrder) – Desired sort order.

• schema (Schema) – Used to resolve each sort field’s source column name; a nested source is rendered as its full dotted path ("addr.city").

Returns:

[] when the orders are identical, otherwise one set_sort_order op with args {"sort_fields": [(column, "asc" | "desc"), ...]}.

Return type:

list[MigrationOp]

Raises:

ValueError – If the orders differ and a target field uses a non-identity transform or a non-default null order (asc must be NULLS_FIRST and desc NULLS_LAST), which MigrationContext.replace_sort_order() cannot express, or a target field’s source id is not in schema.

iceberg_ops.autogen.spec_diff_to_ops(*, current, target, schema)

Compute add / remove ops to evolve current partition spec into target.

Compares fields by their name (the spec-level partition name, not the source column). Each target field absent from current becomes an add_partition_field op; each current field absent from target becomes a remove_partition_field op.

Parameters:

• current (PartitionSpec) – table.spec() from the existing table.

• target (PartitionSpec) – Desired spec.

• schema (Schema) – Used to resolve each partition field’s source column name from its source_id so the generated migration can pass the correct argument to MigrationContext.update_partition_spec(). A nested source is rendered as its full dotted path ("addr.city").

Returns:

Ops in removal-first then add order.

Return type:

list[MigrationOp]

Raises:

• ValueError – If a partition field keeps its name but changes its source column or transform; autogenerate cannot evolve a field in place, so give the new field a different name in the target. Also if an added field’s source id is not in schema.

• Op kinds emitted –

• ---------------- –

• - remove_partition_field - args {"partition_name"} –

• - add_partition_field - args – {"source_column", "transform_source", "partition_name"}, where transform_source is Python source such as "transforms.DayTransform()".

iceberg_ops.autogen.write_autogenerated_revision(*, version, description, ops, target_table, out_path, overwrite=False)

Render and write an autogenerated migration to out_path.

The file is written as UTF-8 with LF line endings on every platform.

Parameters:

• version (str) – Four-digit version, e.g. "0007".

• description (str) – Free-text description of the migration.

• ops (list[MigrationOp]) – Ops to render.

• target_table (str) – Table identifier for the generated calls; prefer a bare name so the migration follows the namespace it runs against.

• out_path (str | Path) – Destination file.

• overwrite (bool, default: False) – Replace an existing file at out_path. Default False.

Returns:

The absolute path of the written file (string for symmetry with iceberg_ops.scaffolding.create_revision()).

Return type:

str

Raises:

• FileExistsError – If out_path exists and overwrite is false.

• ValueError – If version is not four digits or an op has an unknown kind.